2023-04-22 19:53:29,906 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c 2023-04-22 19:53:29,922 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-22 19:53:29,939 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-22 19:53:29,939 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/cluster_cd12d633-6d44-ab55-c2cd-001d95f1c41f, deleteOnExit=true 2023-04-22 19:53:29,940 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-22 19:53:29,941 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/test.cache.data in system properties and HBase conf 2023-04-22 19:53:29,941 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/hadoop.tmp.dir in system properties and HBase conf 2023-04-22 19:53:29,942 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/hadoop.log.dir in system properties and HBase conf 2023-04-22 19:53:29,942 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-22 19:53:29,943 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-22 19:53:29,943 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-22 19:53:30,072 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-22 19:53:30,491 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-22 19:53:30,496 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-22 19:53:30,496 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-22 19:53:30,497 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-22 19:53:30,497 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-22 19:53:30,497 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-22 19:53:30,498 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-22 19:53:30,498 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-22 19:53:30,498 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-22 19:53:30,499 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-22 19:53:30,499 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/nfs.dump.dir in system properties and HBase conf 2023-04-22 19:53:30,500 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/java.io.tmpdir in system properties and HBase conf 2023-04-22 19:53:30,500 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-22 19:53:30,500 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-22 19:53:30,501 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-22 19:53:31,080 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-22 19:53:31,084 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-22 19:53:31,373 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-22 19:53:31,556 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-22 19:53:31,571 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-22 19:53:31,613 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-22 19:53:31,654 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/java.io.tmpdir/Jetty_localhost_40331_hdfs____.d7m4xt/webapp 2023-04-22 19:53:31,798 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40331 2023-04-22 19:53:31,837 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-22 19:53:31,838 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-22 19:53:32,249 WARN [Listener at localhost/32897] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-22 19:53:32,366 WARN [Listener at localhost/32897] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-22 19:53:32,392 WARN [Listener at localhost/32897] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-22 19:53:32,401 INFO [Listener at localhost/32897] log.Slf4jLog(67): jetty-6.1.26 2023-04-22 19:53:32,411 INFO [Listener at localhost/32897] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/java.io.tmpdir/Jetty_localhost_43741_datanode____.i0ue10/webapp 2023-04-22 19:53:32,529 INFO [Listener at localhost/32897] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43741 2023-04-22 19:53:32,958 WARN [Listener at localhost/46665] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-22 19:53:33,568 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xed91cf174ce46f5: Processing first storage report for DS-b06d8ba1-9e6a-48a8-8daf-48b6721f828d from datanode 2596931a-765d-4c8f-8b11-1321bc6bd088 2023-04-22 19:53:33,570 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xed91cf174ce46f5: from storage DS-b06d8ba1-9e6a-48a8-8daf-48b6721f828d node DatanodeRegistration(127.0.0.1:33915, datanodeUuid=2596931a-765d-4c8f-8b11-1321bc6bd088, infoPort=42909, infoSecurePort=0, ipcPort=46665, storageInfo=lv=-57;cid=testClusterID;nsid=1788868982;c=1682193211166), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-22 19:53:33,570 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xed91cf174ce46f5: Processing first storage report for DS-e11f67e2-c2ef-401e-8ffc-f51a4dab9284 from datanode 2596931a-765d-4c8f-8b11-1321bc6bd088 2023-04-22 19:53:33,570 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xed91cf174ce46f5: from storage DS-e11f67e2-c2ef-401e-8ffc-f51a4dab9284 node DatanodeRegistration(127.0.0.1:33915, datanodeUuid=2596931a-765d-4c8f-8b11-1321bc6bd088, infoPort=42909, infoSecurePort=0, ipcPort=46665, storageInfo=lv=-57;cid=testClusterID;nsid=1788868982;c=1682193211166), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-22 19:53:33,625 DEBUG [Listener at localhost/46665] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c 2023-04-22 19:53:33,698 INFO [Listener at localhost/46665] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/cluster_cd12d633-6d44-ab55-c2cd-001d95f1c41f/zookeeper_0, clientPort=53381, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/cluster_cd12d633-6d44-ab55-c2cd-001d95f1c41f/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/cluster_cd12d633-6d44-ab55-c2cd-001d95f1c41f/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-22 19:53:33,711 INFO [Listener at localhost/46665] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=53381 2023-04-22 19:53:33,719 INFO [Listener at localhost/46665] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 19:53:33,722 INFO [Listener at localhost/46665] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 19:53:34,377 INFO [Listener at localhost/46665] util.FSUtils(471): Created version file at hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04 with version=8 2023-04-22 19:53:34,377 INFO [Listener at localhost/46665] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/hbase-staging 2023-04-22 19:53:34,686 INFO [Listener at localhost/46665] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-22 19:53:35,201 INFO [Listener at localhost/46665] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-22 19:53:35,233 INFO [Listener at localhost/46665] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 19:53:35,233 INFO [Listener at localhost/46665] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-22 19:53:35,234 INFO [Listener at localhost/46665] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-22 19:53:35,234 INFO [Listener at localhost/46665] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 19:53:35,234 INFO [Listener at localhost/46665] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-22 19:53:35,374 INFO [Listener at localhost/46665] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-22 19:53:35,445 DEBUG [Listener at localhost/46665] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-22 19:53:35,558 INFO [Listener at localhost/46665] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34929 2023-04-22 19:53:35,568 INFO [Listener at localhost/46665] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 19:53:35,570 INFO [Listener at localhost/46665] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 19:53:35,592 INFO [Listener at localhost/46665] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34929 connecting to ZooKeeper ensemble=127.0.0.1:53381 2023-04-22 19:53:35,633 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:349290x0, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-22 19:53:35,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:34929-0x101ad6f52b90000 connected 2023-04-22 19:53:35,665 DEBUG [Listener at localhost/46665] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-22 19:53:35,666 DEBUG [Listener at localhost/46665] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 19:53:35,670 DEBUG [Listener at localhost/46665] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-22 19:53:35,680 DEBUG [Listener at localhost/46665] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34929 2023-04-22 19:53:35,681 DEBUG [Listener at localhost/46665] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34929 2023-04-22 19:53:35,681 DEBUG [Listener at localhost/46665] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34929 2023-04-22 19:53:35,681 DEBUG [Listener at localhost/46665] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34929 2023-04-22 19:53:35,682 DEBUG [Listener at localhost/46665] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34929 2023-04-22 19:53:35,690 INFO [Listener at localhost/46665] master.HMaster(444): hbase.rootdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04, hbase.cluster.distributed=false 2023-04-22 19:53:35,766 INFO [Listener at localhost/46665] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-22 19:53:35,767 INFO [Listener at localhost/46665] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 19:53:35,767 INFO [Listener at localhost/46665] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-22 19:53:35,767 INFO [Listener at localhost/46665] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-22 19:53:35,767 INFO [Listener at localhost/46665] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 19:53:35,767 INFO [Listener at localhost/46665] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-22 19:53:35,772 INFO [Listener at localhost/46665] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-22 19:53:35,775 INFO [Listener at localhost/46665] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44791 2023-04-22 19:53:35,777 INFO [Listener at localhost/46665] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-22 19:53:35,784 DEBUG [Listener at localhost/46665] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-22 19:53:35,785 INFO [Listener at localhost/46665] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 19:53:35,788 INFO [Listener at localhost/46665] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 19:53:35,791 INFO [Listener at localhost/46665] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44791 connecting to ZooKeeper ensemble=127.0.0.1:53381 2023-04-22 19:53:35,794 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:447910x0, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-22 19:53:35,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:44791-0x101ad6f52b90001 connected 2023-04-22 19:53:35,796 DEBUG [Listener at localhost/46665] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-22 19:53:35,797 DEBUG [Listener at localhost/46665] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 19:53:35,798 DEBUG [Listener at localhost/46665] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-22 19:53:35,799 DEBUG [Listener at localhost/46665] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44791 2023-04-22 19:53:35,800 DEBUG [Listener at localhost/46665] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44791 2023-04-22 19:53:35,807 DEBUG [Listener at localhost/46665] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44791 2023-04-22 19:53:35,808 DEBUG [Listener at localhost/46665] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44791 2023-04-22 19:53:35,808 DEBUG [Listener at localhost/46665] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44791 2023-04-22 19:53:35,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,34929,1682193214523 2023-04-22 19:53:35,820 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-22 19:53:35,822 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,34929,1682193214523 2023-04-22 19:53:35,841 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-22 19:53:35,841 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-22 19:53:35,841 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 19:53:35,842 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-22 19:53:35,843 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,34929,1682193214523 from backup master directory 2023-04-22 19:53:35,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-22 19:53:35,846 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,34929,1682193214523 2023-04-22 19:53:35,846 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-22 19:53:35,847 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-22 19:53:35,847 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,34929,1682193214523 2023-04-22 19:53:35,850 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-22 19:53:35,851 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-22 19:53:35,933 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/hbase.id with ID: b4ef2843-fd22-4eb7-9ccb-7a67be5e71a5 2023-04-22 19:53:35,977 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 19:53:35,991 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 19:53:36,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7b928b0c to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:36,063 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@279de0b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:36,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 19:53:36,092 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-22 19:53:36,111 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-22 19:53:36,111 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-22 19:53:36,113 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:53:36,117 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:53:36,118 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-22 19:53:36,152 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/data/master/store-tmp 2023-04-22 19:53:36,582 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:53:36,582 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-22 19:53:36,582 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 19:53:36,582 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 19:53:36,582 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-22 19:53:36,582 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 19:53:36,582 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 19:53:36,582 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-22 19:53:36,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/WALs/jenkins-hbase4.apache.org,34929,1682193214523 2023-04-22 19:53:36,609 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34929%2C1682193214523, suffix=, logDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/WALs/jenkins-hbase4.apache.org,34929,1682193214523, archiveDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/oldWALs, maxLogs=10 2023-04-22 19:53:36,668 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33915,DS-b06d8ba1-9e6a-48a8-8daf-48b6721f828d,DISK] 2023-04-22 19:53:36,679 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:53:36,740 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/WALs/jenkins-hbase4.apache.org,34929,1682193214523/jenkins-hbase4.apache.org%2C34929%2C1682193214523.1682193216620 2023-04-22 19:53:36,740 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33915,DS-b06d8ba1-9e6a-48a8-8daf-48b6721f828d,DISK]] 2023-04-22 19:53:36,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:53:36,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:53:36,745 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-22 19:53:36,746 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-22 19:53:36,809 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-22 19:53:36,818 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-22 19:53:36,843 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-22 19:53:36,859 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:53:36,866 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-22 19:53:36,868 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-22 19:53:36,892 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-22 19:53:36,897 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 19:53:36,898 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58830932, jitterRate=-0.12335079908370972}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-22 19:53:36,898 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-22 19:53:36,900 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-22 19:53:36,920 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-22 19:53:36,921 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-22 19:53:36,924 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 19:53:36,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-22 19:53:36,960 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-04-22 19:53:36,960 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-22 19:53:36,994 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-22 19:53:37,000 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-22 19:53:37,026 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-22 19:53:37,030 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-22 19:53:37,032 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-22 19:53:37,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-22 19:53:37,044 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-22 19:53:37,049 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 19:53:37,050 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-22 19:53:37,051 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-22 19:53:37,065 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-22 19:53:37,071 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-22 19:53:37,071 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-22 19:53:37,071 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 19:53:37,072 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,34929,1682193214523, sessionid=0x101ad6f52b90000, setting cluster-up flag (Was=false) 2023-04-22 19:53:37,087 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 19:53:37,094 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-22 19:53:37,095 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34929,1682193214523 2023-04-22 19:53:37,101 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 19:53:37,107 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-22 19:53:37,108 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34929,1682193214523 2023-04-22 19:53:37,112 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.hbase-snapshot/.tmp 2023-04-22 19:53:37,213 INFO [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(952): ClusterId : b4ef2843-fd22-4eb7-9ccb-7a67be5e71a5 2023-04-22 19:53:37,218 DEBUG [RS:0;jenkins-hbase4:44791] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-22 19:53:37,225 DEBUG [RS:0;jenkins-hbase4:44791] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-22 19:53:37,225 DEBUG [RS:0;jenkins-hbase4:44791] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-22 19:53:37,229 DEBUG [RS:0;jenkins-hbase4:44791] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-22 19:53:37,230 DEBUG [RS:0;jenkins-hbase4:44791] zookeeper.ReadOnlyZKClient(139): Connect 0x141d15b9 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:37,241 DEBUG [RS:0;jenkins-hbase4:44791] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1adaf94a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:37,241 DEBUG [RS:0;jenkins-hbase4:44791] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ee57ec9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-22 19:53:37,249 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-22 19:53:37,261 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 19:53:37,261 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 19:53:37,261 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 19:53:37,261 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 19:53:37,261 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-22 19:53:37,261 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 19:53:37,262 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-22 19:53:37,262 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 19:53:37,267 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:44791 2023-04-22 19:53:37,268 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682193247268 2023-04-22 19:53:37,271 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-22 19:53:37,273 INFO [RS:0;jenkins-hbase4:44791] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-22 19:53:37,274 INFO [RS:0;jenkins-hbase4:44791] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-22 19:53:37,274 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1023): About to register with Master. 2023-04-22 19:53:37,275 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-22 19:53:37,276 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-22 19:53:37,277 INFO [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,34929,1682193214523 with isa=jenkins-hbase4.apache.org/172.31.14.131:44791, startcode=1682193215766 2023-04-22 19:53:37,281 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-22 19:53:37,283 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-22 19:53:37,291 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-22 19:53:37,291 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-22 19:53:37,292 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-22 19:53:37,292 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-22 19:53:37,296 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:37,305 DEBUG [RS:0;jenkins-hbase4:44791] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-22 19:53:37,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-22 19:53:37,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-22 19:53:37,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-22 19:53:37,318 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-22 19:53:37,318 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-22 19:53:37,321 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682193217321,5,FailOnTimeoutGroup] 2023-04-22 19:53:37,328 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682193217325,5,FailOnTimeoutGroup] 2023-04-22 19:53:37,328 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:37,328 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-22 19:53:37,330 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:37,330 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:37,384 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43989, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-22 19:53:37,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:37,415 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04 2023-04-22 19:53:37,415 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:32897 2023-04-22 19:53:37,416 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-22 19:53:37,422 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 19:53:37,423 DEBUG [RS:0;jenkins-hbase4:44791] zookeeper.ZKUtil(162): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:37,423 WARN [RS:0;jenkins-hbase4:44791] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-22 19:53:37,424 INFO [RS:0;jenkins-hbase4:44791] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-22 19:53:37,425 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1947): logDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/WALs/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:37,427 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,44791,1682193215766] 2023-04-22 19:53:37,437 DEBUG [RS:0;jenkins-hbase4:44791] zookeeper.ZKUtil(162): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:37,451 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-22 19:53:37,460 INFO [RS:0;jenkins-hbase4:44791] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-22 19:53:37,484 INFO [RS:0;jenkins-hbase4:44791] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-22 19:53:37,488 INFO [RS:0;jenkins-hbase4:44791] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-22 19:53:37,488 INFO [RS:0;jenkins-hbase4:44791] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:37,490 INFO [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-22 19:53:37,498 INFO [RS:0;jenkins-hbase4:44791] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:37,498 DEBUG [RS:0;jenkins-hbase4:44791] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 19:53:37,499 DEBUG [RS:0;jenkins-hbase4:44791] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 19:53:37,499 DEBUG [RS:0;jenkins-hbase4:44791] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 19:53:37,499 DEBUG [RS:0;jenkins-hbase4:44791] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 19:53:37,499 DEBUG [RS:0;jenkins-hbase4:44791] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 19:53:37,499 DEBUG [RS:0;jenkins-hbase4:44791] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-22 19:53:37,499 DEBUG [RS:0;jenkins-hbase4:44791] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 19:53:37,499 DEBUG [RS:0;jenkins-hbase4:44791] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 19:53:37,500 DEBUG [RS:0;jenkins-hbase4:44791] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 19:53:37,500 DEBUG [RS:0;jenkins-hbase4:44791] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 19:53:37,501 INFO [RS:0;jenkins-hbase4:44791] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:37,501 INFO [RS:0;jenkins-hbase4:44791] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:37,501 INFO [RS:0;jenkins-hbase4:44791] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:37,519 INFO [RS:0;jenkins-hbase4:44791] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-22 19:53:37,521 INFO [RS:0;jenkins-hbase4:44791] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44791,1682193215766-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:37,540 INFO [RS:0;jenkins-hbase4:44791] regionserver.Replication(203): jenkins-hbase4.apache.org,44791,1682193215766 started 2023-04-22 19:53:37,540 INFO [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,44791,1682193215766, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:44791, sessionid=0x101ad6f52b90001 2023-04-22 19:53:37,540 DEBUG [RS:0;jenkins-hbase4:44791] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-22 19:53:37,540 DEBUG [RS:0;jenkins-hbase4:44791] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:37,540 DEBUG [RS:0;jenkins-hbase4:44791] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44791,1682193215766' 2023-04-22 19:53:37,540 DEBUG [RS:0;jenkins-hbase4:44791] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:37,541 DEBUG [RS:0;jenkins-hbase4:44791] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:37,542 DEBUG [RS:0;jenkins-hbase4:44791] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-22 19:53:37,542 DEBUG [RS:0;jenkins-hbase4:44791] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-22 19:53:37,542 DEBUG [RS:0;jenkins-hbase4:44791] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:37,542 DEBUG [RS:0;jenkins-hbase4:44791] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44791,1682193215766' 2023-04-22 19:53:37,542 DEBUG [RS:0;jenkins-hbase4:44791] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-22 19:53:37,543 DEBUG [RS:0;jenkins-hbase4:44791] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-22 19:53:37,544 DEBUG [RS:0;jenkins-hbase4:44791] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-22 19:53:37,544 INFO [RS:0;jenkins-hbase4:44791] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-22 19:53:37,544 INFO [RS:0;jenkins-hbase4:44791] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-22 19:53:37,657 INFO [RS:0;jenkins-hbase4:44791] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44791%2C1682193215766, suffix=, logDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/WALs/jenkins-hbase4.apache.org,44791,1682193215766, archiveDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/oldWALs, maxLogs=32 2023-04-22 19:53:37,676 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33915,DS-b06d8ba1-9e6a-48a8-8daf-48b6721f828d,DISK] 2023-04-22 19:53:37,682 INFO [RS:0;jenkins-hbase4:44791] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/WALs/jenkins-hbase4.apache.org,44791,1682193215766/jenkins-hbase4.apache.org%2C44791%2C1682193215766.1682193217660 2023-04-22 19:53:37,682 DEBUG [RS:0;jenkins-hbase4:44791] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33915,DS-b06d8ba1-9e6a-48a8-8daf-48b6721f828d,DISK]] 2023-04-22 19:53:37,747 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-22 19:53:37,749 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-22 19:53:37,749 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04 2023-04-22 19:53:38,170 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:53:38,173 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-22 19:53:38,176 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/info 2023-04-22 19:53:38,177 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-22 19:53:38,178 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:53:38,178 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-22 19:53:38,181 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/rep_barrier 2023-04-22 19:53:38,181 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-22 19:53:38,182 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:53:38,182 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-22 19:53:38,185 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/table 2023-04-22 19:53:38,185 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-22 19:53:38,186 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:53:38,188 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740 2023-04-22 19:53:38,189 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740 2023-04-22 19:53:38,193 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 19:53:38,195 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-22 19:53:38,198 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 19:53:38,199 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66273597, jitterRate=-0.01244644820690155}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 19:53:38,199 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-22 19:53:38,200 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-22 19:53:38,200 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-22 19:53:38,200 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-22 19:53:38,200 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-22 19:53:38,200 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-22 19:53:38,201 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-22 19:53:38,201 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-22 19:53:38,206 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-22 19:53:38,206 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-22 19:53:38,214 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-22 19:53:38,229 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-22 19:53:38,232 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-22 19:53:38,384 DEBUG [jenkins-hbase4:34929] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-22 19:53:38,387 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44791,1682193215766, state=OPENING 2023-04-22 19:53:38,395 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-22 19:53:38,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 19:53:38,397 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-22 19:53:38,400 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:53:38,585 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:38,587 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-22 19:53:38,590 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-22 19:53:38,601 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-22 19:53:38,602 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-22 19:53:38,605 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44791%2C1682193215766.meta, suffix=.meta, logDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/WALs/jenkins-hbase4.apache.org,44791,1682193215766, archiveDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/oldWALs, maxLogs=32 2023-04-22 19:53:38,623 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33915,DS-b06d8ba1-9e6a-48a8-8daf-48b6721f828d,DISK] 2023-04-22 19:53:38,627 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/WALs/jenkins-hbase4.apache.org,44791,1682193215766/jenkins-hbase4.apache.org%2C44791%2C1682193215766.meta.1682193218607.meta 2023-04-22 19:53:38,627 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33915,DS-b06d8ba1-9e6a-48a8-8daf-48b6721f828d,DISK]] 2023-04-22 19:53:38,627 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:53:38,630 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-22 19:53:38,645 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-22 19:53:38,650 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-22 19:53:38,655 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-22 19:53:38,655 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:53:38,655 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-22 19:53:38,655 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-22 19:53:38,658 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-22 19:53:38,660 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/info 2023-04-22 19:53:38,660 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/info 2023-04-22 19:53:38,661 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-22 19:53:38,662 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:53:38,662 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-22 19:53:38,663 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/rep_barrier 2023-04-22 19:53:38,663 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/rep_barrier 2023-04-22 19:53:38,664 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-22 19:53:38,665 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:53:38,665 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-22 19:53:38,667 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/table 2023-04-22 19:53:38,667 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/table 2023-04-22 19:53:38,667 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-22 19:53:38,668 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:53:38,670 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740 2023-04-22 19:53:38,673 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740 2023-04-22 19:53:38,676 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 19:53:38,679 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-22 19:53:38,680 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74581205, jitterRate=0.11134655773639679}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 19:53:38,680 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-22 19:53:38,690 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682193218577 2023-04-22 19:53:38,708 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-22 19:53:38,709 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-22 19:53:38,709 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44791,1682193215766, state=OPEN 2023-04-22 19:53:38,712 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-22 19:53:38,712 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-22 19:53:38,717 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-22 19:53:38,717 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44791,1682193215766 in 312 msec 2023-04-22 19:53:38,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-22 19:53:38,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 504 msec 2023-04-22 19:53:38,727 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5560 sec 2023-04-22 19:53:38,727 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682193218727, completionTime=-1 2023-04-22 19:53:38,728 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-22 19:53:38,728 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 19:53:38,795 DEBUG [hconnection-0x7d62d032-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:38,799 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53758, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:38,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-22 19:53:38,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682193278819 2023-04-22 19:53:38,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682193338819 2023-04-22 19:53:38,820 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 91 msec 2023-04-22 19:53:38,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34929,1682193214523-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:38,850 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34929,1682193214523-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:38,850 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34929,1682193214523-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:38,851 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:34929, period=300000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:38,852 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-22 19:53:38,858 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-22 19:53:38,866 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-22 19:53:38,867 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-22 19:53:38,877 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-22 19:53:38,880 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 19:53:38,882 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 19:53:38,907 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/hbase/namespace/163905e55735ce7abf13fc939c703557 2023-04-22 19:53:38,909 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/hbase/namespace/163905e55735ce7abf13fc939c703557 empty. 2023-04-22 19:53:38,910 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/hbase/namespace/163905e55735ce7abf13fc939c703557 2023-04-22 19:53:38,911 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-22 19:53:38,958 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-22 19:53:38,962 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 163905e55735ce7abf13fc939c703557, NAME => 'hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp 2023-04-22 19:53:38,978 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:53:38,979 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 163905e55735ce7abf13fc939c703557, disabling compactions & flushes 2023-04-22 19:53:38,979 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:53:38,979 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:53:38,979 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. after waiting 0 ms 2023-04-22 19:53:38,979 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:53:38,979 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:53:38,979 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 163905e55735ce7abf13fc939c703557: 2023-04-22 19:53:38,985 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 19:53:39,002 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682193218989"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193218989"}]},"ts":"1682193218989"} 2023-04-22 19:53:39,037 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 19:53:39,040 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 19:53:39,045 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193219040"}]},"ts":"1682193219040"} 2023-04-22 19:53:39,050 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-22 19:53:39,061 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=163905e55735ce7abf13fc939c703557, ASSIGN}] 2023-04-22 19:53:39,064 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=163905e55735ce7abf13fc939c703557, ASSIGN 2023-04-22 19:53:39,065 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=163905e55735ce7abf13fc939c703557, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44791,1682193215766; forceNewPlan=false, retain=false 2023-04-22 19:53:39,217 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=163905e55735ce7abf13fc939c703557, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:39,217 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682193219216"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193219216"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193219216"}]},"ts":"1682193219216"} 2023-04-22 19:53:39,223 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 163905e55735ce7abf13fc939c703557, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:53:39,383 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:53:39,385 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 163905e55735ce7abf13fc939c703557, NAME => 'hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557.', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:53:39,385 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 163905e55735ce7abf13fc939c703557 2023-04-22 19:53:39,386 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:53:39,386 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 163905e55735ce7abf13fc939c703557 2023-04-22 19:53:39,386 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 163905e55735ce7abf13fc939c703557 2023-04-22 19:53:39,388 INFO [StoreOpener-163905e55735ce7abf13fc939c703557-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 163905e55735ce7abf13fc939c703557 2023-04-22 19:53:39,390 DEBUG [StoreOpener-163905e55735ce7abf13fc939c703557-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/namespace/163905e55735ce7abf13fc939c703557/info 2023-04-22 19:53:39,391 DEBUG [StoreOpener-163905e55735ce7abf13fc939c703557-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/namespace/163905e55735ce7abf13fc939c703557/info 2023-04-22 19:53:39,391 INFO [StoreOpener-163905e55735ce7abf13fc939c703557-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 163905e55735ce7abf13fc939c703557 columnFamilyName info 2023-04-22 19:53:39,392 INFO [StoreOpener-163905e55735ce7abf13fc939c703557-1] regionserver.HStore(310): Store=163905e55735ce7abf13fc939c703557/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:53:39,394 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/namespace/163905e55735ce7abf13fc939c703557 2023-04-22 19:53:39,395 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/namespace/163905e55735ce7abf13fc939c703557 2023-04-22 19:53:39,400 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 163905e55735ce7abf13fc939c703557 2023-04-22 19:53:39,404 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/namespace/163905e55735ce7abf13fc939c703557/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 19:53:39,404 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 163905e55735ce7abf13fc939c703557; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59134488, jitterRate=-0.1188274621963501}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-22 19:53:39,405 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 163905e55735ce7abf13fc939c703557: 2023-04-22 19:53:39,407 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557., pid=6, masterSystemTime=1682193219377 2023-04-22 19:53:39,411 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:53:39,411 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:53:39,413 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=163905e55735ce7abf13fc939c703557, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:39,413 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682193219412"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682193219412"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682193219412"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682193219412"}]},"ts":"1682193219412"} 2023-04-22 19:53:39,420 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-22 19:53:39,420 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 163905e55735ce7abf13fc939c703557, server=jenkins-hbase4.apache.org,44791,1682193215766 in 195 msec 2023-04-22 19:53:39,424 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-22 19:53:39,425 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=163905e55735ce7abf13fc939c703557, ASSIGN in 359 msec 2023-04-22 19:53:39,426 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 19:53:39,427 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193219427"}]},"ts":"1682193219427"} 2023-04-22 19:53:39,430 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-22 19:53:39,436 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 19:53:39,439 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 568 msec 2023-04-22 19:53:39,480 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-22 19:53:39,482 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-22 19:53:39,482 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 19:53:39,520 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-22 19:53:39,538 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-22 19:53:39,545 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 32 msec 2023-04-22 19:53:39,556 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-22 19:53:39,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-22 19:53:39,576 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-04-22 19:53:39,593 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-22 19:53:39,597 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-22 19:53:39,597 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.750sec 2023-04-22 19:53:39,599 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-22 19:53:39,601 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-22 19:53:39,602 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-22 19:53:39,603 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34929,1682193214523-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-22 19:53:39,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34929,1682193214523-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-22 19:53:39,643 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x69cada93 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:39,649 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-22 19:53:39,649 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@53e64c30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:39,667 DEBUG [hconnection-0x6468bb18-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:39,679 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53766, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:39,689 INFO [Listener at localhost/46665] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,34929,1682193214523 2023-04-22 19:53:39,716 INFO [Listener at localhost/46665] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=359, ProcessCount=169, AvailableMemoryMB=13268 2023-04-22 19:53:39,733 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 19:53:39,737 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53998, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 19:53:39,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 19:53:39,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 19:53:39,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 19:53:39,756 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 19:53:39,759 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 19:53:39,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-22 19:53:39,764 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:39,765 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb empty. 2023-04-22 19:53:39,767 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:39,767 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:53:39,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-22 19:53:39,787 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 19:53:39,790 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d6400a2b7b67145225eb12601c5e92eb, NAME => 'TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp 2023-04-22 19:53:39,801 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:53:39,802 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d6400a2b7b67145225eb12601c5e92eb, disabling compactions & flushes 2023-04-22 19:53:39,802 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:39,802 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:39,802 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. after waiting 0 ms 2023-04-22 19:53:39,802 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:39,802 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:39,802 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:39,806 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 19:53:39,808 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193219808"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193219808"}]},"ts":"1682193219808"} 2023-04-22 19:53:39,811 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 19:53:39,813 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 19:53:39,813 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193219813"}]},"ts":"1682193219813"} 2023-04-22 19:53:39,816 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 19:53:39,821 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6400a2b7b67145225eb12601c5e92eb, ASSIGN}] 2023-04-22 19:53:39,824 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6400a2b7b67145225eb12601c5e92eb, ASSIGN 2023-04-22 19:53:39,826 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6400a2b7b67145225eb12601c5e92eb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44791,1682193215766; forceNewPlan=false, retain=false 2023-04-22 19:53:39,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-22 19:53:39,977 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d6400a2b7b67145225eb12601c5e92eb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:39,978 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193219977"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193219977"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193219977"}]},"ts":"1682193219977"} 2023-04-22 19:53:39,982 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:53:40,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-22 19:53:40,143 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:40,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d6400a2b7b67145225eb12601c5e92eb, NAME => 'TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:53:40,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:40,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:53:40,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:40,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:40,146 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:40,148 DEBUG [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A 2023-04-22 19:53:40,149 DEBUG [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A 2023-04-22 19:53:40,150 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:53:40,151 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6400a2b7b67145225eb12601c5e92eb columnFamilyName A 2023-04-22 19:53:40,152 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] regionserver.HStore(310): Store=d6400a2b7b67145225eb12601c5e92eb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:53:40,152 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:40,154 DEBUG [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B 2023-04-22 19:53:40,155 DEBUG [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B 2023-04-22 19:53:40,155 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:53:40,155 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6400a2b7b67145225eb12601c5e92eb columnFamilyName B 2023-04-22 19:53:40,156 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] regionserver.HStore(310): Store=d6400a2b7b67145225eb12601c5e92eb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:53:40,156 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:40,158 DEBUG [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C 2023-04-22 19:53:40,158 DEBUG [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C 2023-04-22 19:53:40,158 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:53:40,158 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6400a2b7b67145225eb12601c5e92eb columnFamilyName C 2023-04-22 19:53:40,159 INFO [StoreOpener-d6400a2b7b67145225eb12601c5e92eb-1] regionserver.HStore(310): Store=d6400a2b7b67145225eb12601c5e92eb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:53:40,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:40,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:40,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:40,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 19:53:40,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:40,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 19:53:40,171 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d6400a2b7b67145225eb12601c5e92eb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72476166, jitterRate=0.07997903227806091}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 19:53:40,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:40,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., pid=11, masterSystemTime=1682193220136 2023-04-22 19:53:40,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:40,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:40,176 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d6400a2b7b67145225eb12601c5e92eb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:40,176 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193220176"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682193220176"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682193220176"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682193220176"}]},"ts":"1682193220176"} 2023-04-22 19:53:40,183 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-22 19:53:40,183 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 in 198 msec 2023-04-22 19:53:40,186 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-22 19:53:40,186 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6400a2b7b67145225eb12601c5e92eb, ASSIGN in 362 msec 2023-04-22 19:53:40,187 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 19:53:40,188 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193220187"}]},"ts":"1682193220187"} 2023-04-22 19:53:40,190 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 19:53:40,193 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 19:53:40,195 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 442 msec 2023-04-22 19:53:40,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-22 19:53:40,392 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-22 19:53:40,398 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x6d61f291 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:40,403 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f0e2360, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:40,406 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:40,410 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:40,413 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 19:53:40,415 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 19:53:40,423 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x02df095a to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:40,428 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7263b174, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:40,428 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x61e72c30 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:40,434 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@748f869b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:40,435 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:40,439 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56223d42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:40,439 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x4fea27db to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:40,444 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:40,445 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:40,448 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de4d1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:40,449 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x0dbb9ac1 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:40,455 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42927198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:40,455 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x49a45b6d to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:40,484 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1818ad0a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:40,484 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:40,488 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:40,489 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd3abbd to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:53:40,492 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6129209e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:53:40,504 DEBUG [hconnection-0x27e010-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:40,505 DEBUG [hconnection-0x21233b14-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:40,505 DEBUG [hconnection-0x65e031ee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:40,505 DEBUG [hconnection-0x43a25080-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:40,507 DEBUG [hconnection-0x1330534b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:40,508 DEBUG [hconnection-0x65d3daa5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:40,517 DEBUG [hconnection-0x1a5dd2ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:40,518 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:40,518 DEBUG [hconnection-0x3685aa1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:40,519 DEBUG [hconnection-0x5de3642b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:53:40,520 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:40,520 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53790, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:40,524 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53800, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:40,524 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53794, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:40,526 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53802, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:40,526 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53808, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:40,529 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53810, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:40,529 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53822, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:53:40,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:40,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:40,548 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:40,548 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:40,549 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:40,549 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:40,554 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:40,557 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:40,559 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:40,559 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:40,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:40,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:40,561 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:40,561 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:40,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:40,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:40,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:40,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:40,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:40,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:40,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:40,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:40,577 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:40,581 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:40,581 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:40,583 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:40,584 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:40,586 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-22 19:53:40,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:40,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:40,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:40,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:40,608 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:40,608 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:40,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:40,626 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:40,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:40,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:40,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193280713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:40,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193280736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:40,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682193220687, max timestamp=1682193220726Num uniques 859; 2023-04-22 19:53:40,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-22 19:53:40,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682193220687, max timestamp=1682193220726Num uniques 859; 2023-04-22 19:53:40,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-22 19:53:40,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682193220687, max timestamp=1682193220726Num uniques 859; 2023-04-22 19:53:40,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-22 19:53:40,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:40,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193280838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:40,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1682193220818, max timestamp=1682193220835Num uniques 1009; 2023-04-22 19:53:40,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=8194, suffix heap size=132904,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=179 2023-04-22 19:53:40,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193220818, max timestamp=1682193220834Num uniques 1009; 2023-04-22 19:53:40,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-22 19:53:40,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1682193220818, max timestamp=1682193220834Num uniques 1009; 2023-04-22 19:53:40,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=8424, suffix heap size=132904,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=184 2023-04-22 19:53:40,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:40,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:40,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193280917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:40,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193280919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:40,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682193220901, max timestamp=1682193220916Num uniques 1043; 2023-04-22 19:53:40,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=6870, suffix heap size=136440,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=150 2023-04-22 19:53:40,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682193220901, max timestamp=1682193220919Num uniques 1038; 2023-04-22 19:53:40,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=8102, suffix heap size=135920,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=177 2023-04-22 19:53:40,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682193220901, max timestamp=1682193220919Num uniques 1027; 2023-04-22 19:53:40,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 19:53:41,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682193220976, max timestamp=1682193220988Num uniques 1036; 2023-04-22 19:53:41,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682193220980, max timestamp=1682193220988Num uniques 1036; 2023-04-22 19:53:41,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-22 19:53:41,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-22 19:53:41,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682193220976, max timestamp=1682193220997Num uniques 1009; 2023-04-22 19:53:41,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-22 19:53:41,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682193221059, max timestamp=1682193221094Num uniques 1009; 2023-04-22 19:53:41,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13510, suffix heap size=132904,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=295 2023-04-22 19:53:41,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682193221059, max timestamp=1682193221094Num uniques 1009; 2023-04-22 19:53:41,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=13510, suffix heap size=132904,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=295 2023-04-22 19:53:41,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682193221059, max timestamp=1682193221094Num uniques 1009; 2023-04-22 19:53:41,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13510, suffix heap size=132904,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=295 2023-04-22 19:53:41,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193281150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193281153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193281157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193281157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193281165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,179 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/9c85c7a9721f49b198557d5bde210c14 2023-04-22 19:53:41,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193281264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193281264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193281267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193281267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193281270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,303 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/92e7c5db8d88425782c436c1e6dd3aac 2023-04-22 19:53:41,366 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/3801ce1ef3334bcaafef83545c8cce63 2023-04-22 19:53:41,379 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/9c85c7a9721f49b198557d5bde210c14 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9c85c7a9721f49b198557d5bde210c14 2023-04-22 19:53:41,399 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9c85c7a9721f49b198557d5bde210c14, entries=150, sequenceid=10, filesize=11.6 K 2023-04-22 19:53:41,404 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-22 19:53:41,407 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/92e7c5db8d88425782c436c1e6dd3aac as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/92e7c5db8d88425782c436c1e6dd3aac 2023-04-22 19:53:41,448 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/92e7c5db8d88425782c436c1e6dd3aac, entries=150, sequenceid=10, filesize=11.6 K 2023-04-22 19:53:41,454 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/3801ce1ef3334bcaafef83545c8cce63 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3801ce1ef3334bcaafef83545c8cce63 2023-04-22 19:53:41,472 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3801ce1ef3334bcaafef83545c8cce63, entries=150, sequenceid=10, filesize=11.6 K 2023-04-22 19:53:41,479 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for d6400a2b7b67145225eb12601c5e92eb in 892ms, sequenceid=10, compaction requested=false 2023-04-22 19:53:41,481 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-22 19:53:41,485 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:41,485 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:41,485 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:41,486 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:41,486 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,487 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:41,487 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:41,504 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,504 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:41,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:41,505 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,505 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:41,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:41,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:41,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:41,508 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:41,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@339d9870[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:41,508 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=154.31 KB heapSize=432 KB 2023-04-22 19:53:41,511 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:41,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,511 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,511 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:41,511 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:41,511 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:41,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:53:41,511 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:41,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:41,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:41,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:53:41,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:41,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:53:41,519 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:41,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:41,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:41,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:41,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:41,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:41,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:41,533 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:41,533 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:41,533 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:41,533 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:41,533 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:41,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193281514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193281527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193281535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193281540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193281540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,555 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,555 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:41,555 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:41,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,557 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,560 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:41,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:41,583 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:41,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:41,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:41,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:41,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:41,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:41,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:41,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:41,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,586 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:41,586 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:41,626 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/ca8c788a90aa4972a1db1e455600fe6b 2023-04-22 19:53:41,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193281659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193281662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193281662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193281662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193281662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,687 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:41,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:41,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/1bece0325c964ab7acd939ecdac15313 2023-04-22 19:53:41,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:41,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:41,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:41,775 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:41,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:41,779 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:41,779 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:41,781 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,781 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:41,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,783 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,783 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:41,787 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:41,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:41,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:41,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:41,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:41,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:41,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:41,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:41,793 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:41,793 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:41,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:41,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:41,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:41,800 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:41,800 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:41,800 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:41,800 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:41,812 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:41,812 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:41,812 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:41,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:41,822 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:41,823 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:41,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:41,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43daedc5[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:41,827 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:41,827 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,831 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,831 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:41,831 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:41,831 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:41,831 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,832 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:41,841 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/c4db1382afac4193af6564ba8476bd0d 2023-04-22 19:53:41,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,842 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:41,842 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:41,842 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:41,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:41,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:41,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:41,852 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:41,852 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:41,852 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:41,853 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:41,853 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:41,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:41,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:41,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/ca8c788a90aa4972a1db1e455600fe6b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/ca8c788a90aa4972a1db1e455600fe6b 2023-04-22 19:53:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:41,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:41,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:41,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:41,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:41,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:41,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,871 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:41,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:41,870 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:41,871 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:41,872 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:41,871 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:41,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:41,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:41,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193281863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/ca8c788a90aa4972a1db1e455600fe6b, entries=200, sequenceid=116, filesize=13.9 K 2023-04-22 19:53:41,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193281871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193281874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193281874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:41,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193281874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:41,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/1bece0325c964ab7acd939ecdac15313 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/1bece0325c964ab7acd939ecdac15313 2023-04-22 19:53:41,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/1bece0325c964ab7acd939ecdac15313, entries=150, sequenceid=116, filesize=11.6 K 2023-04-22 19:53:41,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/c4db1382afac4193af6564ba8476bd0d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c4db1382afac4193af6564ba8476bd0d 2023-04-22 19:53:41,932 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c4db1382afac4193af6564ba8476bd0d, entries=150, sequenceid=116, filesize=11.6 K 2023-04-22 19:53:41,934 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.95 KB/442320, currentSize=26.84 KB/27480 for d6400a2b7b67145225eb12601c5e92eb in 424ms, sequenceid=116, compaction requested=false 2023-04-22 19:53:41,941 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:41,972 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:41,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:41,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:42,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:42,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:42,004 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:42,004 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:42,013 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:42,013 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:42,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,017 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:42,017 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:42,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:42,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:42,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:42,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:42,031 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:42,043 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:42,043 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:42,043 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:42,044 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:42,045 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-22 19:53:42,046 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:42,046 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:42,046 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:42,046 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:42,046 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:42,046 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:42,112 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/f3182230e77e408bbcdaacfeae2033ee 2023-04-22 19:53:42,192 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/0d4d87daac5a4d7e8275c12f1b239afc 2023-04-22 19:53:42,229 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:42,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:42,239 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/d51cc055c9b54feb87a95fde7b5f7d00 2023-04-22 19:53:42,250 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/f3182230e77e408bbcdaacfeae2033ee as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f3182230e77e408bbcdaacfeae2033ee 2023-04-22 19:53:42,265 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f3182230e77e408bbcdaacfeae2033ee, entries=150, sequenceid=123, filesize=11.6 K 2023-04-22 19:53:42,266 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/0d4d87daac5a4d7e8275c12f1b239afc as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0d4d87daac5a4d7e8275c12f1b239afc 2023-04-22 19:53:42,289 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0d4d87daac5a4d7e8275c12f1b239afc, entries=150, sequenceid=123, filesize=11.6 K 2023-04-22 19:53:42,295 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/d51cc055c9b54feb87a95fde7b5f7d00 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/d51cc055c9b54feb87a95fde7b5f7d00 2023-04-22 19:53:42,321 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/d51cc055c9b54feb87a95fde7b5f7d00, entries=150, sequenceid=123, filesize=11.6 K 2023-04-22 19:53:42,323 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=100.63 KB/103050 for d6400a2b7b67145225eb12601c5e92eb in 278ms, sequenceid=123, compaction requested=true 2023-04-22 19:53:42,323 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:42,323 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:42,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:42,325 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:42,325 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,325 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:42,325 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:42,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:42,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 19:53:42,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:42,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:42,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:42,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:42,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:42,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:42,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,330 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:42,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:42,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:42,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,332 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:42,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:42,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:42,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2afb3472[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:42,336 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:42,336 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,338 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:42,338 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,338 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:42,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:42,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:42,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:42,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:42,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:42,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:42,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:42,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:42,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:42,385 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:42,385 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:42,385 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:42,385 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:42,385 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:42,389 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,389 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:42,389 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:42,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:42,390 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:42,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:42,394 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:42,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:42,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:42,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:42,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:42,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:42,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:42,411 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:42,412 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:42,412 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:42,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,412 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:42,412 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,412 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,412 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,412 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,425 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/d9d3787e205146ebb3ae6ab11b8cbebc 2023-04-22 19:53:42,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193282439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193282445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193282449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193282454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193282451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,510 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/bad04121107948c591582c74826e03a2 2023-04-22 19:53:42,512 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:42,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:42,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:42,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:42,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:42,561 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:42,561 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:42,565 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:42,565 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:42,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,567 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:42,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:42,568 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,569 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,569 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:42,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:42,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:42,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:42,571 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:42,571 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:42,572 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:42,574 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:42,574 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:42,574 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:42,574 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:42,574 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:42,574 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:42,578 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,579 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:42,579 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,579 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:42,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:42,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:42,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:42,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e372d7d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:42,583 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:42,583 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,586 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,586 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:42,586 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:42,586 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:42,588 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,588 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:42,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:42,593 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:42,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:42,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:42,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:42,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:42,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:42,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:42,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:42,612 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:42,612 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:42,612 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:42,612 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:42,612 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:42,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193282596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193282603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:42,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:42,623 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:42,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/97238151514d4ffaad5052995001f9c4 2023-04-22 19:53:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,626 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/d9d3787e205146ebb3ae6ab11b8cbebc as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/d9d3787e205146ebb3ae6ab11b8cbebc 2023-04-22 19:53:42,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193282611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193282611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193282611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,649 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,649 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,649 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,649 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,649 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,649 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:42,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:42,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:42,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:42,649 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,650 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:42,651 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:42,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:42,652 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:42,652 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:42,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:42,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:42,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/d9d3787e205146ebb3ae6ab11b8cbebc, entries=200, sequenceid=142, filesize=14.1 K 2023-04-22 19:53:42,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/bad04121107948c591582c74826e03a2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/bad04121107948c591582c74826e03a2 2023-04-22 19:53:42,695 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/bad04121107948c591582c74826e03a2, entries=150, sequenceid=142, filesize=11.8 K 2023-04-22 19:53:42,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/97238151514d4ffaad5052995001f9c4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/97238151514d4ffaad5052995001f9c4 2023-04-22 19:53:42,716 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/97238151514d4ffaad5052995001f9c4, entries=150, sequenceid=142, filesize=11.8 K 2023-04-22 19:53:42,717 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d6400a2b7b67145225eb12601c5e92eb in 390ms, sequenceid=142, compaction requested=true 2023-04-22 19:53:42,718 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:42,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:42,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:42,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:42,732 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:42,732 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:42,739 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:42,741 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:53:42,741 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:42,742 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/92e7c5db8d88425782c436c1e6dd3aac, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/1bece0325c964ab7acd939ecdac15313, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0d4d87daac5a4d7e8275c12f1b239afc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/bad04121107948c591582c74826e03a2] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=46.6 K 2023-04-22 19:53:42,744 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 92e7c5db8d88425782c436c1e6dd3aac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682193220549 2023-04-22 19:53:42,745 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52424 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:42,745 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1bece0325c964ab7acd939ecdac15313, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682193221151 2023-04-22 19:53:42,747 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:53:42,747 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:42,747 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9c85c7a9721f49b198557d5bde210c14, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/ca8c788a90aa4972a1db1e455600fe6b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f3182230e77e408bbcdaacfeae2033ee, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/d9d3787e205146ebb3ae6ab11b8cbebc] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=51.2 K 2023-04-22 19:53:42,748 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0d4d87daac5a4d7e8275c12f1b239afc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682193221529 2023-04-22 19:53:42,750 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 9c85c7a9721f49b198557d5bde210c14, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682193220549 2023-04-22 19:53:42,753 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting bad04121107948c591582c74826e03a2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682193222311 2023-04-22 19:53:42,752 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:42,758 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting ca8c788a90aa4972a1db1e455600fe6b, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682193221146 2023-04-22 19:53:42,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:42,767 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting f3182230e77e408bbcdaacfeae2033ee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682193221529 2023-04-22 19:53:42,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:42,772 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting d9d3787e205146ebb3ae6ab11b8cbebc, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682193222311 2023-04-22 19:53:42,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:42,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:42,792 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:42,792 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:42,797 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:42,797 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:42,811 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,811 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,813 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:42,813 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:42,814 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,814 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:42,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:42,820 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#13 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:42,826 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#12 average throughput is 0.47 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:42,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:42,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:42,830 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:42,831 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:42,831 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:42,831 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:42,833 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:42,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:42,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:42,835 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 19:53:42,835 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:42,836 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:42,836 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:42,836 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:42,836 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:42,836 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:42,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193282922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193282926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193282927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193282928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:42,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193282928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:42,937 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/27ca87b19f454df48ce0ff84db084e0a 2023-04-22 19:53:42,945 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/22f1e7236a71456f8c497a8440d1636f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/22f1e7236a71456f8c497a8440d1636f 2023-04-22 19:53:42,950 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/082da84dc6ad4a0e9a8114ed538670f2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/082da84dc6ad4a0e9a8114ed538670f2 2023-04-22 19:53:42,980 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into 22f1e7236a71456f8c497a8440d1636f(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:42,981 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:42,981 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=12, startTime=1682193222731; duration=0sec 2023-04-22 19:53:42,982 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:42,982 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:42,985 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:42,985 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/C is initiating minor compaction (all files) 2023-04-22 19:53:42,985 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/C in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:42,985 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3801ce1ef3334bcaafef83545c8cce63, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c4db1382afac4193af6564ba8476bd0d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/d51cc055c9b54feb87a95fde7b5f7d00, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/97238151514d4ffaad5052995001f9c4] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=46.6 K 2023-04-22 19:53:42,986 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 3801ce1ef3334bcaafef83545c8cce63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682193220549 2023-04-22 19:53:42,987 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting c4db1382afac4193af6564ba8476bd0d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682193221151 2023-04-22 19:53:42,988 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting d51cc055c9b54feb87a95fde7b5f7d00, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682193221529 2023-04-22 19:53:43,019 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/641a8e290b0b45c1abbbb1dcf1e88855 2023-04-22 19:53:43,028 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 97238151514d4ffaad5052995001f9c4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682193222311 2023-04-22 19:53:43,031 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into 082da84dc6ad4a0e9a8114ed538670f2(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:43,031 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:43,031 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=12, startTime=1682193222722; duration=0sec 2023-04-22 19:53:43,031 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:43,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193283033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193283033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193283034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193283036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193283043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,056 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#C#compaction#16 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:43,185 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/3a470c83c618495a91a5aa106b6c18bf 2023-04-22 19:53:43,186 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/7b1bdf33392345bc876396da603e09ca as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7b1bdf33392345bc876396da603e09ca 2023-04-22 19:53:43,200 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/C of d6400a2b7b67145225eb12601c5e92eb into 7b1bdf33392345bc876396da603e09ca(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:43,201 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:43,201 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/27ca87b19f454df48ce0ff84db084e0a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/27ca87b19f454df48ce0ff84db084e0a 2023-04-22 19:53:43,201 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/C, priority=12, startTime=1682193222731; duration=0sec 2023-04-22 19:53:43,201 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:43,215 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/27ca87b19f454df48ce0ff84db084e0a, entries=150, sequenceid=156, filesize=11.8 K 2023-04-22 19:53:43,219 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/641a8e290b0b45c1abbbb1dcf1e88855 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/641a8e290b0b45c1abbbb1dcf1e88855 2023-04-22 19:53:43,234 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/641a8e290b0b45c1abbbb1dcf1e88855, entries=150, sequenceid=156, filesize=11.8 K 2023-04-22 19:53:43,244 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/3a470c83c618495a91a5aa106b6c18bf as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3a470c83c618495a91a5aa106b6c18bf 2023-04-22 19:53:43,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193283242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193283242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193283242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193283243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193283246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,279 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3a470c83c618495a91a5aa106b6c18bf, entries=150, sequenceid=156, filesize=11.8 K 2023-04-22 19:53:43,282 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for d6400a2b7b67145225eb12601c5e92eb in 448ms, sequenceid=156, compaction requested=false 2023-04-22 19:53:43,282 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:43,282 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:43,282 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:43,282 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:43,282 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,282 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:43,282 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:43,285 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:43,285 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:43,285 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:43,287 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:43,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:43,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:43,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@213664d3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:43,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:43,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:43,294 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,294 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:43,298 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:43,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:43,299 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:43,299 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:43,299 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:43,301 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:43,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,301 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:43,301 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:43,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:43,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:43,307 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:43,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:43,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:43,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:43,308 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:43,311 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:43,311 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:43,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:43,311 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:43,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:43,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:43,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:43,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:43,315 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:43,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:43,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:43,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:43,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:43,329 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:43,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:43,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:43,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:43,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:43,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:43,332 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,332 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:43,332 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:43,332 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:43,332 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:43,332 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:43,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:43,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:43,433 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:43,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:43,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:43,456 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-22 19:53:43,459 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-22 19:53:43,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:43,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:43,473 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:43,473 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:43,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:43,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:43,478 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:43,478 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:43,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:43,479 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:43,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:43,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:43,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:43,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:43,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:43,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:43,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:43,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:43,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:43,493 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:43,494 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:43,494 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:43,494 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 19:53:43,494 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:43,495 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:43,495 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:43,497 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:43,497 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:43,497 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:43,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:43,587 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:43,588 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/8d3f17b8681e415fbcd4279ba6b2f953 2023-04-22 19:53:43,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193283607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193283609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193283611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193283614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193283614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193283732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193283732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193283734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193283734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193283735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193283944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193283944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193283944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193283945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:43,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:43,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193283946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,077 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/5264011cebfa46a4a1df7073e37722e4 2023-04-22 19:53:44,149 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/c254d8d884b944918b8e9681712ee290 2023-04-22 19:53:44,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/8d3f17b8681e415fbcd4279ba6b2f953 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/8d3f17b8681e415fbcd4279ba6b2f953 2023-04-22 19:53:44,182 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/8d3f17b8681e415fbcd4279ba6b2f953, entries=150, sequenceid=178, filesize=11.8 K 2023-04-22 19:53:44,185 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/5264011cebfa46a4a1df7073e37722e4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5264011cebfa46a4a1df7073e37722e4 2023-04-22 19:53:44,199 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5264011cebfa46a4a1df7073e37722e4, entries=150, sequenceid=178, filesize=11.8 K 2023-04-22 19:53:44,201 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/c254d8d884b944918b8e9681712ee290 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c254d8d884b944918b8e9681712ee290 2023-04-22 19:53:44,239 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c254d8d884b944918b8e9681712ee290, entries=150, sequenceid=178, filesize=11.8 K 2023-04-22 19:53:44,241 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for d6400a2b7b67145225eb12601c5e92eb in 747ms, sequenceid=178, compaction requested=true 2023-04-22 19:53:44,241 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:44,241 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:44,242 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:44,243 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:44,243 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,243 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:44,243 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:44,247 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,247 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:44,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:44,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:44,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@351611f5[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:44,250 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:44,250 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,253 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,253 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,253 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:44,253 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:44,254 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:44,254 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:44,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:44,256 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:44,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:44,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:44,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:44,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:44,268 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-22 19:53:44,270 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 19:53:44,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:44,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:44,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:44,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:44,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:44,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:44,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:44,276 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:44,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:44,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:44,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:44,276 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:44,279 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,279 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,279 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:44,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:44,280 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,283 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:44,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,287 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:44,287 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:44,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:44,289 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:44,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:44,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:44,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:44,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:44,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:44,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:44,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:44,300 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:44,300 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:44,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/cf603db045234b63af0be26f533d5b81 2023-04-22 19:53:44,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193284369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193284373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193284375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193284380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193284383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,400 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:44,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:44,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:44,412 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/de6e4e2affb94378b90d37ccdc7f6f15 2023-04-22 19:53:44,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:44,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:44,422 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:44,422 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:44,425 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:44,425 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:44,425 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,425 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:44,429 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:44,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:44,429 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,430 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,430 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:44,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:44,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:44,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:44,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:44,441 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:44,441 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:44,443 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:44,443 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:44,443 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:44,446 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:44,446 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:44,446 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:44,446 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:44,446 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:44,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:44,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:44,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:44,455 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,455 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:44,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/48f37ef3063740b295fe4d7cf4f60463 2023-04-22 19:53:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:44,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/cf603db045234b63af0be26f533d5b81 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/cf603db045234b63af0be26f533d5b81 2023-04-22 19:53:44,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:44,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@463dfb81[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:44,468 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:44,468 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,469 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,470 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:44,470 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:44,470 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:44,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:44,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:44,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:44,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,479 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/cf603db045234b63af0be26f533d5b81, entries=200, sequenceid=196, filesize=14.1 K 2023-04-22 19:53:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/de6e4e2affb94378b90d37ccdc7f6f15 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/de6e4e2affb94378b90d37ccdc7f6f15 2023-04-22 19:53:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:44,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:44,484 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:44,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:44,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:44,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:44,484 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:44,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193284484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:44,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:44,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:44,489 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:44,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/de6e4e2affb94378b90d37ccdc7f6f15, entries=150, sequenceid=196, filesize=11.8 K 2023-04-22 19:53:44,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193284490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193284490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,497 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/48f37ef3063740b295fe4d7cf4f60463 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/48f37ef3063740b295fe4d7cf4f60463 2023-04-22 19:53:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,507 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:44,507 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:44,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:44,508 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,508 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:44,508 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:44,508 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:44,508 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:44,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:44,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:44,513 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,514 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,514 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,514 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:44,514 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:44,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193284496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193284496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/48f37ef3063740b295fe4d7cf4f60463, entries=150, sequenceid=196, filesize=11.8 K 2023-04-22 19:53:44,526 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for d6400a2b7b67145225eb12601c5e92eb in 258ms, sequenceid=196, compaction requested=true 2023-04-22 19:53:44,526 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:44,529 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:44,529 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:44,529 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:44,529 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:44,529 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:44,532 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48282 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:44,532 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:53:44,532 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:44,533 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/22f1e7236a71456f8c497a8440d1636f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/641a8e290b0b45c1abbbb1dcf1e88855, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5264011cebfa46a4a1df7073e37722e4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/de6e4e2affb94378b90d37ccdc7f6f15] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=47.2 K 2023-04-22 19:53:44,533 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50672 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:44,534 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:53:44,534 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:44,534 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/082da84dc6ad4a0e9a8114ed538670f2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/27ca87b19f454df48ce0ff84db084e0a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/8d3f17b8681e415fbcd4279ba6b2f953, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/cf603db045234b63af0be26f533d5b81] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=49.5 K 2023-04-22 19:53:44,534 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 22f1e7236a71456f8c497a8440d1636f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682193222311 2023-04-22 19:53:44,536 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 082da84dc6ad4a0e9a8114ed538670f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682193222311 2023-04-22 19:53:44,537 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 641a8e290b0b45c1abbbb1dcf1e88855, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682193222391 2023-04-22 19:53:44,538 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 27ca87b19f454df48ce0ff84db084e0a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682193222391 2023-04-22 19:53:44,538 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 5264011cebfa46a4a1df7073e37722e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682193222921 2023-04-22 19:53:44,538 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 8d3f17b8681e415fbcd4279ba6b2f953, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682193222921 2023-04-22 19:53:44,539 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting cf603db045234b63af0be26f533d5b81, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193223611 2023-04-22 19:53:44,540 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting de6e4e2affb94378b90d37ccdc7f6f15, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193223611 2023-04-22 19:53:44,561 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#24 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:44,577 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#25 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:44,614 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:44,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:44,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:44,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:44,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:44,625 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:44,626 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:44,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:44,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:44,628 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,628 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:44,630 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:44,630 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:44,630 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/fa8da5f8ea8c4aa4843b3291488f83f7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/fa8da5f8ea8c4aa4843b3291488f83f7 2023-04-22 19:53:44,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:44,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:44,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:44,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:44,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:44,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:44,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:44,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:44,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:44,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:44,639 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:44,639 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:44,640 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 19:53:44,640 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:44,640 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:44,640 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:44,641 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:44,641 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:44,641 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:44,650 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into fa8da5f8ea8c4aa4843b3291488f83f7(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:44,650 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:44,650 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=12, startTime=1682193224528; duration=0sec 2023-04-22 19:53:44,650 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:44,650 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:44,653 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48282 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:44,654 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/C is initiating minor compaction (all files) 2023-04-22 19:53:44,654 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/C in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:44,654 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7b1bdf33392345bc876396da603e09ca, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3a470c83c618495a91a5aa106b6c18bf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c254d8d884b944918b8e9681712ee290, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/48f37ef3063740b295fe4d7cf4f60463] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=47.2 K 2023-04-22 19:53:44,657 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 7b1bdf33392345bc876396da603e09ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682193222311 2023-04-22 19:53:44,658 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 3a470c83c618495a91a5aa106b6c18bf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682193222391 2023-04-22 19:53:44,658 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c254d8d884b944918b8e9681712ee290, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682193222921 2023-04-22 19:53:44,664 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 48f37ef3063740b295fe4d7cf4f60463, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193223611 2023-04-22 19:53:44,682 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-22 19:53:44,691 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-22 19:53:44,692 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#C#compaction#27 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:44,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:44,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:44,707 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/1e3b13be5b6843cf9d0327ec82d916e6 2023-04-22 19:53:44,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193284738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193284741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193284744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193284744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193284745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,752 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/25bb682bdb8a4d3dba3c330345d48774 2023-04-22 19:53:44,772 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/fef0509ad33a4fa9951209ae2c8c001c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/fef0509ad33a4fa9951209ae2c8c001c 2023-04-22 19:53:44,789 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/C of d6400a2b7b67145225eb12601c5e92eb into fef0509ad33a4fa9951209ae2c8c001c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:44,789 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:44,789 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/C, priority=12, startTime=1682193224529; duration=0sec 2023-04-22 19:53:44,789 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:44,855 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/5f99f7ce28234219964fc91e853b75c7 2023-04-22 19:53:44,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193284859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,867 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/1e3b13be5b6843cf9d0327ec82d916e6 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1e3b13be5b6843cf9d0327ec82d916e6 2023-04-22 19:53:44,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193284860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193284862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193284862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:44,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193284863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,878 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1e3b13be5b6843cf9d0327ec82d916e6, entries=150, sequenceid=211, filesize=11.8 K 2023-04-22 19:53:44,894 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/25bb682bdb8a4d3dba3c330345d48774 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/25bb682bdb8a4d3dba3c330345d48774 2023-04-22 19:53:44,910 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/25bb682bdb8a4d3dba3c330345d48774, entries=150, sequenceid=211, filesize=11.8 K 2023-04-22 19:53:44,918 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/5f99f7ce28234219964fc91e853b75c7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/5f99f7ce28234219964fc91e853b75c7 2023-04-22 19:53:44,954 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/5f99f7ce28234219964fc91e853b75c7, entries=150, sequenceid=211, filesize=11.8 K 2023-04-22 19:53:44,969 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for d6400a2b7b67145225eb12601c5e92eb in 330ms, sequenceid=211, compaction requested=false 2023-04-22 19:53:44,969 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:44,969 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:44,969 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:44,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:44,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:44,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:44,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,977 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:44,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:44,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:44,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a67afe[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:44,979 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:44,979 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,981 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:44,981 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:44,981 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:44,981 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:44,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:44,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:44,983 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:44,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:44,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:44,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:44,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:44,992 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:44,992 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:44,992 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:44,992 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:44,992 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:44,995 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,995 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,995 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:44,996 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:44,999 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,005 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:45,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:45,005 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,005 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,010 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:45,010 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:45,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:45,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:45,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:45,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:45,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:45,023 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/ead8ccd4e9414f9c9ad01aed4b5bf706 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/ead8ccd4e9414f9c9ad01aed4b5bf706 2023-04-22 19:53:45,042 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into ead8ccd4e9414f9c9ad01aed4b5bf706(size=12.0 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:45,042 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:45,044 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=12, startTime=1682193224529; duration=0sec 2023-04-22 19:53:45,044 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:45,071 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 19:53:45,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:45,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:45,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:45,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:45,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:45,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:45,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:45,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193285087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193285090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193285092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193285094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193285095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,110 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:45,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/f1fb8e9b5663431ca6b2580f451cb17c 2023-04-22 19:53:45,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:45,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:45,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:45,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:45,142 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:45,142 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:45,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:45,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:45,151 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,151 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,153 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:45,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,154 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,155 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:45,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:45,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:45,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:45,159 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:45,159 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:45,160 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:45,164 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:45,164 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:45,164 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:45,164 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:45,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:45,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:45,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,170 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:45,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@132496e5[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:45,174 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:45,174 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,176 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,176 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:45,178 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,179 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:45,179 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:45,179 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:45,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:45,197 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:45,197 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:45,197 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:45,197 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:45,197 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:45,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193285197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,201 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,201 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:45,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,201 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:45,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193285199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193285200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193285200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193285201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,210 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,213 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,216 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:45,216 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:45,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:45,216 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,216 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:45,217 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:45,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:45,217 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:45,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:45,217 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,217 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:45,217 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,217 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,217 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:45,217 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:45,318 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:45,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:45,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:45,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:45,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:45,336 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:45,336 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:45,338 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:45,338 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:45,340 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,340 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,341 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:45,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:45,342 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,342 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,342 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:45,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:45,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:45,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:45,346 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:45,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:45,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:45,347 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:45,347 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:45,347 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:45,347 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:45,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:45,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:45,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:45,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:45,351 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:45,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:45,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16b8a4cd[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:45,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:45,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,356 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,357 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:45,357 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:45,357 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:45,357 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,357 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:45,360 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:45,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,360 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,360 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:45,372 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:45,372 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:45,372 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:45,372 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:45,372 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:45,374 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,375 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:45,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,375 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,377 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,380 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,386 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,386 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,386 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:45,386 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:45,387 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:45,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:45,387 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,386 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:45,387 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,387 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,387 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:45,387 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:45,387 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:45,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:45,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193285404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193285404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193285407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193285408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193285410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,487 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:45,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:45,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:45,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:45,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:45,494 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:45,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:45,496 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:45,496 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:45,496 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,496 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,498 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:45,498 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:45,499 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,499 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:45,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:45,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:45,504 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:45,504 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:45,505 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:45,507 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:45,507 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:45,507 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:45,507 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,509 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,510 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,510 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:45,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f70a468[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:45,513 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:45,513 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,515 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,515 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:45,515 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:45,515 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,515 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:45,515 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:45,518 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:45,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:45,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:45,526 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:45,526 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:45,526 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:45,526 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:45,526 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:45,528 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,528 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,528 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:45,530 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,533 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,536 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:45,536 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:45,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:45,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:45,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:45,537 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:45,537 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:45,537 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:45,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,546 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:45,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:45,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/f2ddf535203b4832bef69842d7a187a0 2023-04-22 19:53:45,638 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:45,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:45,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:45,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:45,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:45,644 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:45,644 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:45,645 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:45,645 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:45,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,648 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:45,648 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:45,648 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,648 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:45,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:45,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:45,650 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:45,650 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:45,651 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:45,651 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:45,651 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:45,652 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:45,652 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:45,652 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:45,653 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:45,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:45,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:45,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:45,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:45,658 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,659 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,659 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,659 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:45,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d3348f2[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:45,662 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:45,663 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,664 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,665 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:45,665 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:45,665 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:45,666 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,666 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:45,669 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:45,669 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,669 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:45,669 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:45,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:45,677 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:45,677 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:45,677 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:45,677 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:45,677 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:45,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:45,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,682 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,686 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,687 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,687 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:45,687 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:45,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:45,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:45,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:45,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:45,687 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:45,687 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,689 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:45,689 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,689 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:45,689 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,689 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,689 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:45,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193285712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193285712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193285713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193285714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:45,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193285716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,789 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:45,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:45,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:45,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:45,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:45,799 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:45,799 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:45,801 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:45,801 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:45,801 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,801 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,803 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:45,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:45,803 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:45,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:45,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:45,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:45,821 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:45,835 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:45,835 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:45,835 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:45,835 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:45,835 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:45,835 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:45,835 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,835 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:45,835 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:45,837 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,837 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:45,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a17740f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:45,841 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:45,841 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:45,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:45,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:45,843 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,843 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:45,846 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,846 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:45,847 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:45,847 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:45,855 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:45,855 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:45,855 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:45,855 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:45,855 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:45,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:45,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:45,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:45,858 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,860 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:45,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:45,863 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:45,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:45,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:45,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:45,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:45,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:45,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:45,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,865 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:45,865 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:45,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:45,965 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:45,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:45,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:45,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:45,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:45,994 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:45,994 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:45,997 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:45,997 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:45,998 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:45,998 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:46,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:46,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:46,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:46,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:46,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:46,032 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:46,032 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:46,032 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/b07566910d704fe3bc46cdc9e574212c 2023-04-22 19:53:46,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/f1fb8e9b5663431ca6b2580f451cb17c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f1fb8e9b5663431ca6b2580f451cb17c 2023-04-22 19:53:46,044 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,044 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:46,044 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:46,044 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:46,047 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:46,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69da214c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:46,049 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:46,050 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f1fb8e9b5663431ca6b2580f451cb17c, entries=200, sequenceid=233, filesize=14.1 K 2023-04-22 19:53:46,051 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,055 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:46,055 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:46,055 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:46,056 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/f2ddf535203b4832bef69842d7a187a0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/f2ddf535203b4832bef69842d7a187a0 2023-04-22 19:53:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:46,058 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:46,065 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:46,065 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:46,065 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:46,065 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:46,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:46,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:46,066 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:46,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/f2ddf535203b4832bef69842d7a187a0, entries=150, sequenceid=233, filesize=11.8 K 2023-04-22 19:53:46,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,068 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,069 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:46,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/b07566910d704fe3bc46cdc9e574212c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b07566910d704fe3bc46cdc9e574212c 2023-04-22 19:53:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,071 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,080 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,080 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,080 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,080 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,080 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,081 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:46,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:46,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:46,082 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:46,082 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:46,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b07566910d704fe3bc46cdc9e574212c, entries=150, sequenceid=233, filesize=11.8 K 2023-04-22 19:53:46,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:46,086 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d6400a2b7b67145225eb12601c5e92eb in 1016ms, sequenceid=233, compaction requested=true 2023-04-22 19:53:46,087 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:46,088 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:46,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:46,091 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:46,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:46,092 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:46,094 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:46,094 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38772 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:46,096 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:53:46,096 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:53:46,096 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,096 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,096 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/ead8ccd4e9414f9c9ad01aed4b5bf706, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/25bb682bdb8a4d3dba3c330345d48774, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/f2ddf535203b4832bef69842d7a187a0] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=35.5 K 2023-04-22 19:53:46,096 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/fa8da5f8ea8c4aa4843b3291488f83f7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1e3b13be5b6843cf9d0327ec82d916e6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f1fb8e9b5663431ca6b2580f451cb17c] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=37.9 K 2023-04-22 19:53:46,097 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ead8ccd4e9414f9c9ad01aed4b5bf706, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193223611 2023-04-22 19:53:46,100 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting fa8da5f8ea8c4aa4843b3291488f83f7, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193223611 2023-04-22 19:53:46,102 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 25bb682bdb8a4d3dba3c330345d48774, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1682193224322 2023-04-22 19:53:46,102 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 1e3b13be5b6843cf9d0327ec82d916e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1682193224322 2023-04-22 19:53:46,105 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f2ddf535203b4832bef69842d7a187a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682193224738 2023-04-22 19:53:46,105 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting f1fb8e9b5663431ca6b2580f451cb17c, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682193224713 2023-04-22 19:53:46,126 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#33 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:46,139 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#34 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:46,184 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:46,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:46,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:46,190 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/684321cbf38c445b948a61f2a9810520 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/684321cbf38c445b948a61f2a9810520 2023-04-22 19:53:46,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:46,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:46,204 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:46,204 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:46,209 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:46,209 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:46,210 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,210 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,217 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,218 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,218 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,218 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:46,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:46,225 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into 684321cbf38c445b948a61f2a9810520(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:46,225 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:46,225 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=13, startTime=1682193226091; duration=0sec 2023-04-22 19:53:46,226 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:46,226 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:46,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:46,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 19:53:46,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:46,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:46,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:46,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:46,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:46,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:46,231 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:46,231 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/C is initiating minor compaction (all files) 2023-04-22 19:53:46,231 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/C in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,232 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/fef0509ad33a4fa9951209ae2c8c001c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/5f99f7ce28234219964fc91e853b75c7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b07566910d704fe3bc46cdc9e574212c] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=35.5 K 2023-04-22 19:53:46,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:46,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:46,232 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:46,234 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:46,235 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:46,235 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:46,235 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,235 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting fef0509ad33a4fa9951209ae2c8c001c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193223611 2023-04-22 19:53:46,237 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:46,237 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 5f99f7ce28234219964fc91e853b75c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1682193224322 2023-04-22 19:53:46,238 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:46,238 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,239 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:46,239 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:46,239 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,239 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b07566910d704fe3bc46cdc9e574212c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682193224738 2023-04-22 19:53:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,251 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:46,251 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:46,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53b39bfb[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:46,254 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:46,254 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,260 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:46,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:46,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:46,264 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,264 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:46,265 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,266 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:46,266 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,266 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193286266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193286269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193286270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193286271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193286279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,288 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#C#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:46,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:46,292 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:46,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:46,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:46,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:46,292 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:46,295 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,295 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,295 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:46,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,299 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,304 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:46,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:46,304 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,304 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,304 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,323 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,323 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,323 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:46,324 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:46,324 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:46,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:46,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/56ee2bc4e4ad48318fec22ff0d4e607a 2023-04-22 19:53:46,348 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/070674eaf90a412caeb261f6dde3fff3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/070674eaf90a412caeb261f6dde3fff3 2023-04-22 19:53:46,362 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/C of d6400a2b7b67145225eb12601c5e92eb into 070674eaf90a412caeb261f6dde3fff3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:46,362 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:46,362 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/C, priority=13, startTime=1682193226092; duration=0sec 2023-04-22 19:53:46,362 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:46,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193286393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193286394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193286394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193286394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193286396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,424 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:46,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:46,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:46,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:46,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:46,432 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:46,433 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:46,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:46,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:46,435 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,435 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,439 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,439 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,440 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,440 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:46,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:46,441 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:46,441 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:46,445 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:46,445 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:46,445 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:46,445 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,445 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:46,445 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:46,446 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,446 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:46,446 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:46,446 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,446 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:46,446 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:46,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,448 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:46,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44d03da7[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:46,451 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:46,451 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,452 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:46,453 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:46,452 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,453 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:46,453 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:46,454 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:46,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:46,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:46,464 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:46,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:46,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:46,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:46,464 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:46,466 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,466 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,466 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,467 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,472 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,473 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:46,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:46,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:46,475 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:46,475 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:46,575 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:46,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:46,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:46,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:46,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:46,587 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:46,587 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:46,589 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:46,589 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:46,589 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,589 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,591 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,591 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:46,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:46,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:46,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:46,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:46,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:46,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:46,595 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:46,596 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,596 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:46,596 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:46,596 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:46,599 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,600 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:46,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193286598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193286599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193286601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193286602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193286603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:46,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e68e3e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:46,606 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:46,606 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,608 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,609 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:46,609 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,609 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:46,609 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:46,609 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:46,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:46,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:46,612 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:46,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:46,618 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:46,618 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:46,618 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:46,618 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:46,618 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:46,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,622 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:46,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:46,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:46,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,631 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:46,632 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/427d1bcae0fc488481aab28986f8d9a3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/427d1bcae0fc488481aab28986f8d9a3 2023-04-22 19:53:46,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:46,632 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:46,648 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into 427d1bcae0fc488481aab28986f8d9a3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:46,648 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:46,648 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=13, startTime=1682193226087; duration=0sec 2023-04-22 19:53:46,648 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:46,737 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:46,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:46,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:46,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:46,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:46,748 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:46,748 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:46,749 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:46,749 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:46,751 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,751 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,755 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,756 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,757 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,757 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:46,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:46,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:46,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:46,765 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,766 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:46,766 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:46,766 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:46,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:46,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:46,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:46,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,768 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,768 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:46,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d8e078f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:46,772 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:46,772 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,773 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:46,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:46,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:46,774 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,774 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:46,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:46,781 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:46,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:46,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:46,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:46,782 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:46,787 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,788 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,788 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,789 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,791 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,794 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,794 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/09881ea805334fcea821bc95e098d300 2023-04-22 19:53:46,795 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,795 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:46,797 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,797 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,797 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,797 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,797 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,797 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:46,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:46,799 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:46,799 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:46,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:46,899 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:46,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:46,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:46,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:46,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:46,905 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:46,905 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:46,909 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:46,909 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:46,909 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,909 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193286904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,911 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193286904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,912 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,912 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193286905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193286909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:46,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193286910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:46,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:46,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:46,915 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:46,915 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:46,916 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:46,916 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:46,916 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:46,916 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:46,916 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:46,916 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:46,918 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,918 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@322d2786[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:46,922 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:46,922 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,926 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:46,926 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,926 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:46,928 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:46,928 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,928 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,928 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:46,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:46,937 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:46,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:46,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:46,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:46,937 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:46,939 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,939 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,939 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:46,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,940 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,943 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,946 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,946 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,946 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:46,946 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:46,946 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:46,946 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:46,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:46,947 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:46,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:46,946 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:46,949 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:46,949 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:46,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:46,949 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:46,949 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:46,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:47,050 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:47,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:47,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:47,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:47,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:47,060 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:47,060 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:47,071 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:47,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:47,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,075 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,075 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:47,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:47,075 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,076 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:47,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:47,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:47,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:47,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:47,086 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:47,087 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:47,087 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:47,087 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:47,087 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:47,087 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:47,087 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:47,087 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:47,087 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:47,087 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,088 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:47,088 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:47,089 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,090 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,090 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,090 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:47,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d50961[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:47,093 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:47,093 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,094 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:47,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:47,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:47,095 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,095 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:47,097 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:47,097 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:47,097 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,097 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:47,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:47,102 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:47,102 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:47,102 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:47,102 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:47,102 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:47,104 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:47,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:47,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,108 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,111 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:47,111 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:47,111 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,111 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,111 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,111 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:47,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:47,111 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,115 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,115 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,115 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:47,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:47,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:47,116 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:47,116 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:47,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:47,216 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:47,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:47,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:47,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:47,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:47,221 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:47,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:47,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:47,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:47,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,222 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,223 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:47,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:47,223 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,223 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:47,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:47,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:47,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:47,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:47,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:47,225 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:47,225 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:47,225 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:47,225 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:47,225 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:47,225 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:47,225 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:47,225 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:47,225 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,225 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:47,225 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:47,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/7cdae021a3034378a84f609c07e5902e 2023-04-22 19:53:47,227 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,227 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,227 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:47,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:47,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b3eedb[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:47,231 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:47,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,234 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,234 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:47,235 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:47,235 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:47,235 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,235 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:47,236 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,236 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/56ee2bc4e4ad48318fec22ff0d4e607a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/56ee2bc4e4ad48318fec22ff0d4e607a 2023-04-22 19:53:47,236 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:47,237 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:47,237 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:47,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:47,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:47,242 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:47,242 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:47,242 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:47,242 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:47,242 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:47,245 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,245 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,245 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:47,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,246 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,248 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/56ee2bc4e4ad48318fec22ff0d4e607a, entries=150, sequenceid=248, filesize=11.8 K 2023-04-22 19:53:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/09881ea805334fcea821bc95e098d300 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/09881ea805334fcea821bc95e098d300 2023-04-22 19:53:47,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,255 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:47,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:47,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:47,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:47,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:47,256 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,256 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:47,256 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:47,256 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:47,256 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:47,256 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,259 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/09881ea805334fcea821bc95e098d300, entries=150, sequenceid=248, filesize=11.8 K 2023-04-22 19:53:47,261 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/7cdae021a3034378a84f609c07e5902e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7cdae021a3034378a84f609c07e5902e 2023-04-22 19:53:47,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7cdae021a3034378a84f609c07e5902e, entries=150, sequenceid=248, filesize=11.8 K 2023-04-22 19:53:47,274 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d6400a2b7b67145225eb12601c5e92eb in 1048ms, sequenceid=248, compaction requested=false 2023-04-22 19:53:47,274 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:47,356 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:47,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:47,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:47,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:47,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:47,365 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:47,365 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:47,365 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:47,365 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:47,366 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,366 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,367 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:47,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:47,367 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:47,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:47,369 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:47,369 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:47,369 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,369 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:47,369 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:47,371 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:47,371 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:47,371 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 19:53:47,372 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:47,372 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:47,372 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:47,372 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:47,372 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:47,372 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:47,394 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/2ddb5cbf91f046edaaddd1158c62b3a5 2023-04-22 19:53:47,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:47,422 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:47,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193287435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193287436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193287439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193287440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193287440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,457 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/65fc155c669f4639a8445a583b84f3f9 2023-04-22 19:53:47,488 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/89711432d3a0497fb9c4f3fce9bfddc0 2023-04-22 19:53:47,496 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/2ddb5cbf91f046edaaddd1158c62b3a5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/2ddb5cbf91f046edaaddd1158c62b3a5 2023-04-22 19:53:47,503 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/2ddb5cbf91f046edaaddd1158c62b3a5, entries=150, sequenceid=268, filesize=11.9 K 2023-04-22 19:53:47,505 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/65fc155c669f4639a8445a583b84f3f9 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/65fc155c669f4639a8445a583b84f3f9 2023-04-22 19:53:47,525 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/65fc155c669f4639a8445a583b84f3f9, entries=150, sequenceid=268, filesize=11.9 K 2023-04-22 19:53:47,526 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/89711432d3a0497fb9c4f3fce9bfddc0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/89711432d3a0497fb9c4f3fce9bfddc0 2023-04-22 19:53:47,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193287542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,549 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/89711432d3a0497fb9c4f3fce9bfddc0, entries=150, sequenceid=268, filesize=11.9 K 2023-04-22 19:53:47,558 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d6400a2b7b67145225eb12601c5e92eb in 187ms, sequenceid=268, compaction requested=true 2023-04-22 19:53:47,558 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:47,558 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:47,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:47,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:47,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:47,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,561 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 19:53:47,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:47,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:47,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:47,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:47,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:47,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:47,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:47,573 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:47,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b0c33fd[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:47,576 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:47,576 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,577 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:47,578 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,578 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:47,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:47,588 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:47,588 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:47,588 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:47,588 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:47,588 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:47,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193287585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193287586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193287586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193287587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,593 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,593 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,594 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:47,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,597 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,599 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:47,599 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,599 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:47,599 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:47,599 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,600 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,600 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,599 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:47,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:47,600 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:47,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:47,600 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,600 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:47,600 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:47,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193287692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193287693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193287693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193287694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,700 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:47,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:47,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:47,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:47,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:47,707 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:47,708 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:47,708 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:47,708 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:47,710 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,710 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,711 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,711 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:47,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,713 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,713 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:47,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:47,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:47,714 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:47,714 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:47,715 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:47,715 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:47,715 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:47,715 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:47,715 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,715 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:47,715 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:47,718 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:47,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50c6240d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:47,720 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:47,720 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,721 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,721 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:47,725 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:47,725 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:47,725 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:47,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:47,726 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:47,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:47,729 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:47,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:47,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:47,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:47,729 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:47,731 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,731 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:47,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,731 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,732 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,735 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:47,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:47,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:47,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:47,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,744 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:47,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:47,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:47,744 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:47,745 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:47,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193287760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,845 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:47,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:47,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:47,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:47,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:47,850 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:47,850 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:47,850 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:47,850 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:47,851 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,851 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,852 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:47,852 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:47,853 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,853 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:47,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:47,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:47,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:47,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:47,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:47,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:47,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:47,854 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:47,854 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:47,855 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:47,855 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:47,856 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:47,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f87b175[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:47,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:47,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,860 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,861 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:47,861 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:47,861 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:47,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:47,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:47,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:47,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:47,867 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,867 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:47,867 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:47,867 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:47,867 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:47,867 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:47,867 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:47,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:47,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,871 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,872 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,880 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,880 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,880 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:47,880 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,880 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,880 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:47,880 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:47,880 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:47,880 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:47,880 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:47,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:47,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:47,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:47,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:47,881 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:47,881 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:47,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:47,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193287897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193287898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193287898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193287899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:47,980 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/1caca770e1e445eabd364f48d73f10ae 2023-04-22 19:53:47,981 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:47,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:47,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:48,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:48,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:48,004 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:48,004 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,004 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:48,004 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:48,005 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,005 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,007 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,008 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,008 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:48,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:48,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:48,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:48,013 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,013 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:48,016 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,016 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:48,016 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:48,016 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:48,030 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:48,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@165b5368[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:48,034 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:48,034 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,036 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:48,036 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,036 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:48,038 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:48,038 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,038 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,038 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:48,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:48,042 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:48,042 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:48,042 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:48,042 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:48,042 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:48,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,046 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,048 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,052 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,052 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,053 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:48,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:48,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,054 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:48,054 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:48,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:48,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53810 deadline: 1682193288065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,154 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:48,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:48,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:48,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:48,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:48,160 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:48,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:48,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:48,161 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,161 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,162 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,163 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,163 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:48,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:48,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:48,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:48,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:48,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c220985[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:48,173 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:48,173 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:48,176 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,176 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:48,178 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:48,180 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:48,180 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:48,180 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:48,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:48,181 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:48,182 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,182 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,182 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,183 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,186 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,188 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,188 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,188 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,188 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,189 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,188 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,189 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,189 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,189 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:48,189 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:48,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,189 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:48,189 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:48,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193288201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193288201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193288203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193288203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,289 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:48,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:48,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:48,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:48,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:48,301 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:48,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:48,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:48,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,305 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,305 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,305 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:48,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:48,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:48,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:48,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:48,314 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:48,314 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:48,316 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,316 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:48,317 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:48,317 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:48,320 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,320 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,321 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,321 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:48,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1da5ac40[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:48,324 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:48,324 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,326 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,326 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:48,326 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:48,326 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:48,326 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,326 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:48,328 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:48,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:48,331 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:48,331 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:48,331 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:48,331 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:48,331 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:48,334 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,334 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,335 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,335 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,337 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,340 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,340 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,340 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:48,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:48,340 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,341 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,341 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,341 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,341 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,341 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:48,341 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:48,340 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,341 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:48,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/e2404d8178234ec98b4d8db3b116318d 2023-04-22 19:53:48,441 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:48,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:48,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:48,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:48,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:48,447 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:48,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:48,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:48,448 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,448 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,449 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,450 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:48,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:48,451 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:48,451 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,451 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:48,455 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:48,456 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:48,456 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:48,456 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:48,456 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:48,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/233bbf70d5b54bb899ddbbacda9b1329 2023-04-22 19:53:48,461 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:48,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f1bba98[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:48,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:48,465 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,466 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:48,466 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,466 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:48,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/1caca770e1e445eabd364f48d73f10ae as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1caca770e1e445eabd364f48d73f10ae 2023-04-22 19:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:48,472 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:48,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:48,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:48,473 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:48,473 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:48,475 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,475 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,476 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1caca770e1e445eabd364f48d73f10ae, entries=200, sequenceid=284, filesize=14.3 K 2023-04-22 19:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,476 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,477 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/e2404d8178234ec98b4d8db3b116318d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e2404d8178234ec98b4d8db3b116318d 2023-04-22 19:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,479 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:48,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:48,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,483 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,484 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:48,484 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:48,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e2404d8178234ec98b4d8db3b116318d, entries=150, sequenceid=284, filesize=11.9 K 2023-04-22 19:53:48,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/233bbf70d5b54bb899ddbbacda9b1329 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/233bbf70d5b54bb899ddbbacda9b1329 2023-04-22 19:53:48,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/233bbf70d5b54bb899ddbbacda9b1329, entries=150, sequenceid=284, filesize=11.9 K 2023-04-22 19:53:48,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for d6400a2b7b67145225eb12601c5e92eb in 932ms, sequenceid=284, compaction requested=true 2023-04-22 19:53:48,493 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:48,494 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:48,494 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:48,494 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:48,494 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:48,494 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:48,498 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51260 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:48,498 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:53:48,498 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,498 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/427d1bcae0fc488481aab28986f8d9a3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/56ee2bc4e4ad48318fec22ff0d4e607a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/2ddb5cbf91f046edaaddd1158c62b3a5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1caca770e1e445eabd364f48d73f10ae] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=50.1 K 2023-04-22 19:53:48,498 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48820 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:48,498 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:53:48,498 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,499 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/684321cbf38c445b948a61f2a9810520, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/09881ea805334fcea821bc95e098d300, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/65fc155c669f4639a8445a583b84f3f9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e2404d8178234ec98b4d8db3b116318d] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=47.7 K 2023-04-22 19:53:48,499 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 684321cbf38c445b948a61f2a9810520, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682193224738 2023-04-22 19:53:48,500 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 09881ea805334fcea821bc95e098d300, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682193225084 2023-04-22 19:53:48,501 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 427d1bcae0fc488481aab28986f8d9a3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682193224738 2023-04-22 19:53:48,502 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 65fc155c669f4639a8445a583b84f3f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682193226265 2023-04-22 19:53:48,503 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 56ee2bc4e4ad48318fec22ff0d4e607a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682193225084 2023-04-22 19:53:48,503 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 2ddb5cbf91f046edaaddd1158c62b3a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682193226265 2023-04-22 19:53:48,503 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting e2404d8178234ec98b4d8db3b116318d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682193227438 2023-04-22 19:53:48,504 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 1caca770e1e445eabd364f48d73f10ae, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682193227434 2023-04-22 19:53:48,522 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#45 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:48,526 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#46 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:48,553 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/e48b1f11a3594038a001cc6e44b54aa7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/e48b1f11a3594038a001cc6e44b54aa7 2023-04-22 19:53:48,564 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into e48b1f11a3594038a001cc6e44b54aa7(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:48,564 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:48,564 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=12, startTime=1682193228494; duration=0sec 2023-04-22 19:53:48,565 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:48,565 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:48,565 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/dc05e87821e84dce93f82a8dd65446d8 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/dc05e87821e84dce93f82a8dd65446d8 2023-04-22 19:53:48,567 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48820 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:48,568 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/C is initiating minor compaction (all files) 2023-04-22 19:53:48,568 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/C in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,568 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/070674eaf90a412caeb261f6dde3fff3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7cdae021a3034378a84f609c07e5902e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/89711432d3a0497fb9c4f3fce9bfddc0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/233bbf70d5b54bb899ddbbacda9b1329] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=47.7 K 2023-04-22 19:53:48,569 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 070674eaf90a412caeb261f6dde3fff3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682193224738 2023-04-22 19:53:48,569 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 7cdae021a3034378a84f609c07e5902e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682193225084 2023-04-22 19:53:48,570 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 89711432d3a0497fb9c4f3fce9bfddc0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682193226265 2023-04-22 19:53:48,571 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 233bbf70d5b54bb899ddbbacda9b1329, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682193227438 2023-04-22 19:53:48,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:48,588 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:48,588 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 19:53:48,589 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:48,589 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into dc05e87821e84dce93f82a8dd65446d8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:48,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:48,589 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:48,589 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:48,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:48,589 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:48,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:48,589 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=12, startTime=1682193228494; duration=0sec 2023-04-22 19:53:48,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:48,590 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:48,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:48,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:48,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:48,598 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:48,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:48,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:48,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,600 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,600 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,602 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,602 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:48,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:48,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:48,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:48,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:48,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:48,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,614 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#C#compaction#48 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:48,616 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63977c00[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:48,621 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:48,621 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,623 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,623 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:48,623 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,623 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:48,623 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:48,623 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:48,625 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:48,626 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,626 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:48,626 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:48,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:48,631 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:48,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:48,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:48,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:48,631 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:48,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193288631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,634 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,634 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,634 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,635 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,636 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,639 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,639 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,639 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,639 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:48,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:48,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:48,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,639 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,639 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,639 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,639 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:48,639 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:48,639 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,639 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,640 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193288707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193288708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193288708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193288708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193288735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,739 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:48,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:48,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:48,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:48,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:48,744 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:48,744 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,744 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:48,744 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:48,745 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,745 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,747 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,747 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,747 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,747 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:48,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:48,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:48,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:48,748 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:48,748 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:48,749 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,749 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:48,749 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:48,749 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:48,751 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,751 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,751 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,751 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f952fd6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:48,753 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:48,753 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,754 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:48,754 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,754 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:48,755 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:48,755 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:48,756 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:48,759 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:48,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:48,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:48,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:48,759 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:48,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,768 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,769 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:48,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:48,783 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:48,783 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:48,883 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:48,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:48,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:48,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:48,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:48,889 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:48,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:48,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:48,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,891 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,891 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,891 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,891 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:48,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:48,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:48,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:48,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:48,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:48,896 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,896 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:48,896 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:48,896 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:48,898 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f6255c8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:48,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:48,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,905 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,905 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:48,905 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:48,906 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,906 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:48,907 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:48,910 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:48,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:48,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:48,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:48,910 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:48,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:48,912 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:48,913 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:48,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:48,919 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:48,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:48,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:48,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:48,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:48,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:48,923 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:48,924 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:48,924 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:48,924 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:48,924 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,924 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:48,924 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:48,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:48,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:48,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:48,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193288937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,007 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/bf84ac2ac7c748d98c1d940897a1cc7b 2023-04-22 19:53:49,024 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:49,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:49,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/d0deefc0e7bf41d0b44ac3d4282cf90a 2023-04-22 19:53:49,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:49,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:49,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:49,030 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:49,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:49,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:49,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:49,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:49,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:49,032 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:49,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:49,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:49,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:49,035 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:49,035 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:49,035 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:49,041 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/4876531ed8dc4da7a6a84d91bb0dd60b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/4876531ed8dc4da7a6a84d91bb0dd60b 2023-04-22 19:53:49,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:49,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:49,048 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:49,048 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:49,048 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:49,048 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:49,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:49,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:49,053 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/C of d6400a2b7b67145225eb12601c5e92eb into 4876531ed8dc4da7a6a84d91bb0dd60b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:49,053 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:49,053 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/C, priority=12, startTime=1682193228494; duration=0sec 2023-04-22 19:53:49,053 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:49,056 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:49,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:49,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ddff7c6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:49,062 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:49,062 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,064 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,064 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:49,064 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:49,065 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,065 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:49,068 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/b541c3230bdb410aac4f4d31ef2156d7 2023-04-22 19:53:49,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:49,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:49,073 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:49,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:49,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:49,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:49,073 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:49,075 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,075 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:49,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,075 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:49,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:49,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/bf84ac2ac7c748d98c1d940897a1cc7b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/bf84ac2ac7c748d98c1d940897a1cc7b 2023-04-22 19:53:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,080 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,086 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,089 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:49,089 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:49,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:49,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,090 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:49,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:49,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:49,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:49,090 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:49,090 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:49,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:49,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:49,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/bf84ac2ac7c748d98c1d940897a1cc7b, entries=200, sequenceid=304, filesize=14.3 K 2023-04-22 19:53:49,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/d0deefc0e7bf41d0b44ac3d4282cf90a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d0deefc0e7bf41d0b44ac3d4282cf90a 2023-04-22 19:53:49,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d0deefc0e7bf41d0b44ac3d4282cf90a, entries=150, sequenceid=304, filesize=11.9 K 2023-04-22 19:53:49,108 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/b541c3230bdb410aac4f4d31ef2156d7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b541c3230bdb410aac4f4d31ef2156d7 2023-04-22 19:53:49,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b541c3230bdb410aac4f4d31ef2156d7, entries=150, sequenceid=304, filesize=11.9 K 2023-04-22 19:53:49,118 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d6400a2b7b67145225eb12601c5e92eb in 530ms, sequenceid=304, compaction requested=false 2023-04-22 19:53:49,118 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:49,190 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:49,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:49,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:49,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:49,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:49,195 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:49,196 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:49,196 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:49,196 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:49,196 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,196 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:49,197 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:49,198 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,198 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:49,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:49,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:49,200 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:49,200 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:49,200 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 19:53:49,200 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:49,200 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:49,200 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:49,200 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:49,201 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:49,201 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:49,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:49,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:49,246 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/1fbf739a289f4b3ca20082aad2721167 2023-04-22 19:53:49,264 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/b8e16d0401fc4642992e282798780688 2023-04-22 19:53:49,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:49,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193289315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:49,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193289420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:49,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193289624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:49,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193289715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:49,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193289716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:49,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193289722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:49,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193289723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,726 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/bda9dcebfb1446c2ba23f198e09e931b 2023-04-22 19:53:49,735 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/1fbf739a289f4b3ca20082aad2721167 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1fbf739a289f4b3ca20082aad2721167 2023-04-22 19:53:49,756 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1fbf739a289f4b3ca20082aad2721167, entries=150, sequenceid=319, filesize=11.9 K 2023-04-22 19:53:49,758 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/b8e16d0401fc4642992e282798780688 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b8e16d0401fc4642992e282798780688 2023-04-22 19:53:49,765 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b8e16d0401fc4642992e282798780688, entries=150, sequenceid=319, filesize=11.9 K 2023-04-22 19:53:49,767 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/bda9dcebfb1446c2ba23f198e09e931b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/bda9dcebfb1446c2ba23f198e09e931b 2023-04-22 19:53:49,774 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/bda9dcebfb1446c2ba23f198e09e931b, entries=150, sequenceid=319, filesize=11.9 K 2023-04-22 19:53:49,776 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d6400a2b7b67145225eb12601c5e92eb in 576ms, sequenceid=319, compaction requested=true 2023-04-22 19:53:49,776 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:49,776 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:49,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:49,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:49,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:49,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15f0e501[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:49,783 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:49,783 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,789 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:49,792 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:49,792 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:49,793 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:49,793 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:49,793 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:49,793 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:49,793 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:49,793 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:49,794 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:49,794 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,794 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:49,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:49,795 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:49,796 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,800 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,800 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,800 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,801 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:49,801 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:49,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:49,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:49,801 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:49,801 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:49,801 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:49,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:49,802 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:49,802 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,802 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:49,802 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:49,902 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:49,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:49,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:49,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:49,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:49,907 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:49,907 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:49,907 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:49,907 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:49,908 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,908 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:49,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,911 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,911 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:49,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:49,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:49,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:49,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:49,927 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:49,927 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:49,927 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:49,928 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:49,928 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:49,928 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:49,928 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:49,928 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 19:53:49,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:49,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:49,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:49,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:49,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:49,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:49,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:49,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:49,960 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/6521f931af4e4f3dae67b718c852e339 2023-04-22 19:53:49,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:49,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193289968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:49,983 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/9c1fbe2ef79242f3bc52ee2dfe600ec0 2023-04-22 19:53:50,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:50,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193290076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:50,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193290278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,409 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/87767bd4d27d40339fca18a463373450 2023-04-22 19:53:50,416 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/6521f931af4e4f3dae67b718c852e339 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6521f931af4e4f3dae67b718c852e339 2023-04-22 19:53:50,423 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6521f931af4e4f3dae67b718c852e339, entries=200, sequenceid=338, filesize=14.3 K 2023-04-22 19:53:50,424 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/9c1fbe2ef79242f3bc52ee2dfe600ec0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/9c1fbe2ef79242f3bc52ee2dfe600ec0 2023-04-22 19:53:50,436 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/9c1fbe2ef79242f3bc52ee2dfe600ec0, entries=150, sequenceid=338, filesize=11.9 K 2023-04-22 19:53:50,438 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/87767bd4d27d40339fca18a463373450 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/87767bd4d27d40339fca18a463373450 2023-04-22 19:53:50,445 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/87767bd4d27d40339fca18a463373450, entries=150, sequenceid=338, filesize=11.9 K 2023-04-22 19:53:50,446 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d6400a2b7b67145225eb12601c5e92eb in 518ms, sequenceid=338, compaction requested=true 2023-04-22 19:53:50,446 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:50,446 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:50,446 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:50,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:50,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:50,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:50,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:50,526 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:50,526 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:50,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:50,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@333d83f6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:50,528 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:50,528 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:50,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:50,585 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 19:53:50,585 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:50,586 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:50,586 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:50,586 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:50,586 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:50,586 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:50,594 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:50,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:50,594 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,594 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:50,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:50,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193290641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:50,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193290744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:50,856 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:50,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:50,861 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:50,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:50,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:50,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:50,862 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:50,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:50,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:50,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:50,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:50,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:50,912 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:50,924 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:50,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:50,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193290947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:51,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:51,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:51,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:51,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:51,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:51,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:51,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:51,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:51,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:51,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:51,003 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:51,003 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:51,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:51,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/65a1e10706df43e5ae9e84fc3e487b3a 2023-04-22 19:53:51,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/e89d9b1d198a4364b1cca0845f59948f 2023-04-22 19:53:51,048 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/c2e97959ad164b57baa5b475e5bf241c 2023-04-22 19:53:51,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/65a1e10706df43e5ae9e84fc3e487b3a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/65a1e10706df43e5ae9e84fc3e487b3a 2023-04-22 19:53:51,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/65a1e10706df43e5ae9e84fc3e487b3a, entries=100, sequenceid=352, filesize=9.5 K 2023-04-22 19:53:51,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/e89d9b1d198a4364b1cca0845f59948f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e89d9b1d198a4364b1cca0845f59948f 2023-04-22 19:53:51,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e89d9b1d198a4364b1cca0845f59948f, entries=100, sequenceid=352, filesize=9.5 K 2023-04-22 19:53:51,073 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/c2e97959ad164b57baa5b475e5bf241c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c2e97959ad164b57baa5b475e5bf241c 2023-04-22 19:53:51,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c2e97959ad164b57baa5b475e5bf241c, entries=100, sequenceid=352, filesize=9.5 K 2023-04-22 19:53:51,082 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d6400a2b7b67145225eb12601c5e92eb in 497ms, sequenceid=352, compaction requested=true 2023-04-22 19:53:51,082 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:51,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:51,082 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:53:51,082 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:53:51,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:51,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:51,084 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63878 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:53:51,084 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:53:51,084 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58998 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:53:51,084 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:51,084 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:53:51,084 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/e48b1f11a3594038a001cc6e44b54aa7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/bf84ac2ac7c748d98c1d940897a1cc7b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1fbf739a289f4b3ca20082aad2721167, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6521f931af4e4f3dae67b718c852e339, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/65a1e10706df43e5ae9e84fc3e487b3a] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=62.4 K 2023-04-22 19:53:51,084 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:51,085 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/dc05e87821e84dce93f82a8dd65446d8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d0deefc0e7bf41d0b44ac3d4282cf90a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b8e16d0401fc4642992e282798780688, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/9c1fbe2ef79242f3bc52ee2dfe600ec0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e89d9b1d198a4364b1cca0845f59948f] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=57.6 K 2023-04-22 19:53:51,088 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting dc05e87821e84dce93f82a8dd65446d8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682193227438 2023-04-22 19:53:51,088 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting e48b1f11a3594038a001cc6e44b54aa7, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682193227438 2023-04-22 19:53:51,093 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting d0deefc0e7bf41d0b44ac3d4282cf90a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682193227570 2023-04-22 19:53:51,093 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting bf84ac2ac7c748d98c1d940897a1cc7b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682193227570 2023-04-22 19:53:51,093 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b8e16d0401fc4642992e282798780688, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682193228622 2023-04-22 19:53:51,093 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 1fbf739a289f4b3ca20082aad2721167, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682193228622 2023-04-22 19:53:51,094 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 9c1fbe2ef79242f3bc52ee2dfe600ec0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1682193229287 2023-04-22 19:53:51,094 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting e89d9b1d198a4364b1cca0845f59948f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682193229942 2023-04-22 19:53:51,095 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 6521f931af4e4f3dae67b718c852e339, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1682193229287 2023-04-22 19:53:51,096 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 65a1e10706df43e5ae9e84fc3e487b3a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682193229942 2023-04-22 19:53:51,103 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:51,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:51,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:51,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:51,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:51,120 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:51,120 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:51,120 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:51,120 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:51,121 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,121 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:51,126 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#60 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:51,138 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#61 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:51,175 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/5816344e8b4d4588b8ef246fe865854f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5816344e8b4d4588b8ef246fe865854f 2023-04-22 19:53:51,178 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/6405866964b846f2ae35d2ed92587632 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/6405866964b846f2ae35d2ed92587632 2023-04-22 19:53:51,183 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into 5816344e8b4d4588b8ef246fe865854f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:51,184 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:51,184 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=11, startTime=1682193231082; duration=0sec 2023-04-22 19:53:51,184 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:51,184 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:53:51,186 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58998 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:53:51,186 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/C is initiating minor compaction (all files) 2023-04-22 19:53:51,186 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:51,186 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/C in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:51,186 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into 6405866964b846f2ae35d2ed92587632(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:51,186 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,186 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/4876531ed8dc4da7a6a84d91bb0dd60b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b541c3230bdb410aac4f4d31ef2156d7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/bda9dcebfb1446c2ba23f198e09e931b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/87767bd4d27d40339fca18a463373450, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c2e97959ad164b57baa5b475e5bf241c] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=57.6 K 2023-04-22 19:53:51,186 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:51,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:51,187 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=11, startTime=1682193231082; duration=0sec 2023-04-22 19:53:51,187 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:51,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:51,187 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 4876531ed8dc4da7a6a84d91bb0dd60b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682193227438 2023-04-22 19:53:51,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,188 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting b541c3230bdb410aac4f4d31ef2156d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682193227570 2023-04-22 19:53:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:51,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:51,189 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting bda9dcebfb1446c2ba23f198e09e931b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682193228622 2023-04-22 19:53:51,189 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 87767bd4d27d40339fca18a463373450, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1682193229287 2023-04-22 19:53:51,190 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c2e97959ad164b57baa5b475e5bf241c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682193229942 2023-04-22 19:53:51,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:51,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:51,191 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:51,191 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:51,191 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 19:53:51,191 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:51,191 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:51,192 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:51,192 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:51,192 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:51,192 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:51,205 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/15e1a54631064394bcfb9f97db9516ae 2023-04-22 19:53:51,206 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#C#compaction#63 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:51,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:51,255 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:51,280 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/5d283e1e607d4d0bad8c286267d31219 2023-04-22 19:53:51,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:51,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193291305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:51,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193291410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:51,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193291613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,654 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/1e7a87fe871742b2b5972dd38aff1091 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1e7a87fe871742b2b5972dd38aff1091 2023-04-22 19:53:51,661 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/C of d6400a2b7b67145225eb12601c5e92eb into 1e7a87fe871742b2b5972dd38aff1091(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:51,661 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:51,661 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/C, priority=11, startTime=1682193231082; duration=0sec 2023-04-22 19:53:51,661 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:51,716 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/0fd5c38ebb91470e8b5f93d817a32e81 2023-04-22 19:53:51,724 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/15e1a54631064394bcfb9f97db9516ae as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/15e1a54631064394bcfb9f97db9516ae 2023-04-22 19:53:51,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:51,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193291728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,731 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:53:51,733 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/15e1a54631064394bcfb9f97db9516ae, entries=150, sequenceid=372, filesize=11.9 K 2023-04-22 19:53:51,735 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/5d283e1e607d4d0bad8c286267d31219 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5d283e1e607d4d0bad8c286267d31219 2023-04-22 19:53:51,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:51,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:51,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193291734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193291736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:51,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193291736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,739 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:53:51,740 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:53:51,740 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:53:51,744 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5d283e1e607d4d0bad8c286267d31219, entries=150, sequenceid=372, filesize=11.9 K 2023-04-22 19:53:51,745 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/0fd5c38ebb91470e8b5f93d817a32e81 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0fd5c38ebb91470e8b5f93d817a32e81 2023-04-22 19:53:51,756 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0fd5c38ebb91470e8b5f93d817a32e81, entries=150, sequenceid=372, filesize=11.9 K 2023-04-22 19:53:51,757 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d6400a2b7b67145225eb12601c5e92eb in 566ms, sequenceid=372, compaction requested=false 2023-04-22 19:53:51,757 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:51,757 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:51,759 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,759 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:51,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e5c4c4c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:51,762 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:51,762 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:51,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:51,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:51,763 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,763 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:51,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:51,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:51,774 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:51,774 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:51,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:51,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:51,775 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:51,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:51,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:51,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:51,778 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,780 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:51,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:51,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:51,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:51,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:51,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:51,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:51,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:51,786 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:51,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:51,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:51,787 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:51,787 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:51,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:51,887 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:51,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:51,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:51,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:51,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:51,892 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:51,892 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:51,892 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:51,892 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:51,892 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,892 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:51,894 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:51,894 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:51,894 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:51,894 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:51,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:51,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:51,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:51,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:51,897 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:51,897 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:51,897 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 19:53:51,898 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:51,898 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:51,898 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:51,898 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:51,898 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:51,898 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:51,910 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/6bdc6e679f104db8a56fa9ffe0e5266a 2023-04-22 19:53:51,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:51,921 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:51,960 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/21c5b2d7ea6d4c23923fbbc4de7c7efd 2023-04-22 19:53:52,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:52,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193292006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:52,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193292110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:52,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193292313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,402 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/0a5263ac8ed74d938a199dfab7d38e5a 2023-04-22 19:53:52,412 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/6bdc6e679f104db8a56fa9ffe0e5266a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6bdc6e679f104db8a56fa9ffe0e5266a 2023-04-22 19:53:52,418 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6bdc6e679f104db8a56fa9ffe0e5266a, entries=150, sequenceid=387, filesize=11.9 K 2023-04-22 19:53:52,420 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/21c5b2d7ea6d4c23923fbbc4de7c7efd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/21c5b2d7ea6d4c23923fbbc4de7c7efd 2023-04-22 19:53:52,426 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/21c5b2d7ea6d4c23923fbbc4de7c7efd, entries=150, sequenceid=387, filesize=11.9 K 2023-04-22 19:53:52,428 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/0a5263ac8ed74d938a199dfab7d38e5a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0a5263ac8ed74d938a199dfab7d38e5a 2023-04-22 19:53:52,439 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0a5263ac8ed74d938a199dfab7d38e5a, entries=150, sequenceid=387, filesize=11.9 K 2023-04-22 19:53:52,440 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d6400a2b7b67145225eb12601c5e92eb in 543ms, sequenceid=387, compaction requested=true 2023-04-22 19:53:52,441 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:52,441 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:52,441 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:52,441 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:52,441 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,443 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:52,443 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:52,448 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:52,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:52,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:52,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:52,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c0fe986[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:52,452 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:52,452 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:52,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:52,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:52,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:52,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:52,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:52,455 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:52,455 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:52,455 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:52,460 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:52,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:52,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:52,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:52,460 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:52,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:52,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:52,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:52,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:52,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:52,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:52,466 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,474 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,474 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:52,474 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:52,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:52,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:52,474 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:52,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:52,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:52,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:52,474 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:52,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:52,475 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:52,475 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,475 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:52,475 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:52,475 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:52,475 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:52,575 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:52,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:52,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:52,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:52,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:52,584 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:52,584 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:52,584 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:52,584 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:52,589 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:52,589 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:52,591 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:52,591 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:52,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:52,593 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:52,593 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:52,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:52,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:52,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:52,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:52,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:52,594 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:52,594 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:52,594 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 19:53:52,594 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:52,594 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:52,594 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:52,594 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:52,595 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:52,595 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:52,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:52,618 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:52,624 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/78a35cf9fc2d4e699291b09cfbaab63e 2023-04-22 19:53:52,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:52,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193292669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:52,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193292773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:52,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:52,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193292976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,052 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/d012860eded04c17a9959a07d73bb0af 2023-04-22 19:53:53,091 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/2ef56d4f783346a29a9a4d8635e3c9a7 2023-04-22 19:53:53,099 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/78a35cf9fc2d4e699291b09cfbaab63e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/78a35cf9fc2d4e699291b09cfbaab63e 2023-04-22 19:53:53,106 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/78a35cf9fc2d4e699291b09cfbaab63e, entries=150, sequenceid=405, filesize=11.9 K 2023-04-22 19:53:53,108 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/d012860eded04c17a9959a07d73bb0af as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d012860eded04c17a9959a07d73bb0af 2023-04-22 19:53:53,117 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d012860eded04c17a9959a07d73bb0af, entries=150, sequenceid=405, filesize=11.9 K 2023-04-22 19:53:53,119 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/2ef56d4f783346a29a9a4d8635e3c9a7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2ef56d4f783346a29a9a4d8635e3c9a7 2023-04-22 19:53:53,131 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2ef56d4f783346a29a9a4d8635e3c9a7, entries=150, sequenceid=405, filesize=11.9 K 2023-04-22 19:53:53,132 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d6400a2b7b67145225eb12601c5e92eb in 538ms, sequenceid=405, compaction requested=true 2023-04-22 19:53:53,132 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:53,132 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,132 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:53,132 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:53,132 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,133 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:53,133 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:53,134 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,135 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,135 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:53,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec5d9eb[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:53,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:53,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,140 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:53,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:53,148 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:53,157 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:53,157 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:53,157 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:53,157 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:53,157 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:53,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,161 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:53,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:53,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,172 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:53,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:53,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,173 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:53,173 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:53,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:53,273 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:53,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:53,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:53,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:53,284 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 19:53:53,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:53,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:53,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:53,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:53,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:53,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:53,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:53,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:53,308 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:53,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:53,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:53,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:53,309 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,309 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,311 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,311 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:53,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:53,311 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,312 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:53,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:53,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:53,332 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/79ce90e6974347299db7d076317f0ada 2023-04-22 19:53:53,333 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,333 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:53,334 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:53,334 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,336 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:53,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b8cff6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:53,341 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:53,341 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,343 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:53,343 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:53,343 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:53,343 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:53,345 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:53,348 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:53,348 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:53,348 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:53,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:53,349 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:53,352 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,352 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,352 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,353 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,355 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,358 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:53,358 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:53,359 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,359 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:53,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:53,359 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:53,359 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:53,359 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,359 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,359 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,359 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,359 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,359 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:53,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:53,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193293364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/3f77722a84fe422bb54030587af86566 2023-04-22 19:53:53,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/58e7f21ea0f24898a219348f1eceac6c 2023-04-22 19:53:53,432 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/79ce90e6974347299db7d076317f0ada as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/79ce90e6974347299db7d076317f0ada 2023-04-22 19:53:53,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/79ce90e6974347299db7d076317f0ada, entries=200, sequenceid=420, filesize=14.3 K 2023-04-22 19:53:53,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/3f77722a84fe422bb54030587af86566 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/3f77722a84fe422bb54030587af86566 2023-04-22 19:53:53,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/3f77722a84fe422bb54030587af86566, entries=150, sequenceid=420, filesize=11.9 K 2023-04-22 19:53:53,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/58e7f21ea0f24898a219348f1eceac6c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/58e7f21ea0f24898a219348f1eceac6c 2023-04-22 19:53:53,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/58e7f21ea0f24898a219348f1eceac6c, entries=150, sequenceid=420, filesize=11.9 K 2023-04-22 19:53:53,455 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d6400a2b7b67145225eb12601c5e92eb in 171ms, sequenceid=420, compaction requested=true 2023-04-22 19:53:53,455 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:53,456 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:53,456 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:53:53,456 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:53:53,456 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:53,456 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:53,459 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:53,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:53,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:53,465 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64052 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:53:53,465 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:53:53,465 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,465 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5816344e8b4d4588b8ef246fe865854f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/15e1a54631064394bcfb9f97db9516ae, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6bdc6e679f104db8a56fa9ffe0e5266a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/78a35cf9fc2d4e699291b09cfbaab63e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/79ce90e6974347299db7d076317f0ada] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=62.6 K 2023-04-22 19:53:53,465 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:53:53,466 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:53:53,466 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,466 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/6405866964b846f2ae35d2ed92587632, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5d283e1e607d4d0bad8c286267d31219, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/21c5b2d7ea6d4c23923fbbc4de7c7efd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d012860eded04c17a9959a07d73bb0af, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/3f77722a84fe422bb54030587af86566] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=60.2 K 2023-04-22 19:53:53,467 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 5816344e8b4d4588b8ef246fe865854f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682193229927 2023-04-22 19:53:53,467 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 6405866964b846f2ae35d2ed92587632, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682193229927 2023-04-22 19:53:53,467 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 15e1a54631064394bcfb9f97db9516ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1682193230631 2023-04-22 19:53:53,467 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 5d283e1e607d4d0bad8c286267d31219, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1682193230631 2023-04-22 19:53:53,468 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 6bdc6e679f104db8a56fa9ffe0e5266a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682193231267 2023-04-22 19:53:53,468 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 21c5b2d7ea6d4c23923fbbc4de7c7efd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682193231267 2023-04-22 19:53:53,469 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting d012860eded04c17a9959a07d73bb0af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682193231994 2023-04-22 19:53:53,469 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 78a35cf9fc2d4e699291b09cfbaab63e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682193231994 2023-04-22 19:53:53,469 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 79ce90e6974347299db7d076317f0ada, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1682193232627 2023-04-22 19:53:53,469 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 3f77722a84fe422bb54030587af86566, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1682193232627 2023-04-22 19:53:53,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:53,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 19:53:53,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:53,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:53,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:53,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:53,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:53,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:53,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:53,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:53,484 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:53,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:53,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:53,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:53,485 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,485 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,486 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:53,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,488 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:53,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:53,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:53,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:53,493 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:53,493 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:53,493 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:53,495 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:53,498 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:53,498 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:53,498 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,498 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:53,498 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:53,498 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,498 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:53,498 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:53,498 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,498 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:53,498 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:53,500 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,500 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,500 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#77 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,500 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,501 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:53,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d7ad1a1[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:53,503 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:53,503 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,506 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:53,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:53,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:53,506 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,507 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:53,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:53,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,508 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:53,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:53,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:53,512 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:53,512 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:53,512 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:53,512 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:53,512 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:53,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/c276f28f3efb4ffdad9419c0612cbbc3 2023-04-22 19:53:53,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,522 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,525 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:53,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:53,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:53,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:53,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:53,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,525 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:53,525 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:53,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:53,536 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/5e0ca88f7b3e4cc69aafaa52d2f0fbe1 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5e0ca88f7b3e4cc69aafaa52d2f0fbe1 2023-04-22 19:53:53,545 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into 5e0ca88f7b3e4cc69aafaa52d2f0fbe1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:53,545 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:53,546 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=11, startTime=1682193233456; duration=0sec 2023-04-22 19:53:53,546 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:53,546 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:53:53,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:53,547 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/b4abd2104a224688b157df112b83d600 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b4abd2104a224688b157df112b83d600 2023-04-22 19:53:53,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193293536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,550 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:53:53,550 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/C is initiating minor compaction (all files) 2023-04-22 19:53:53,550 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/C in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,550 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1e7a87fe871742b2b5972dd38aff1091, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0fd5c38ebb91470e8b5f93d817a32e81, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0a5263ac8ed74d938a199dfab7d38e5a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2ef56d4f783346a29a9a4d8635e3c9a7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/58e7f21ea0f24898a219348f1eceac6c] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=60.2 K 2023-04-22 19:53:53,551 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 1e7a87fe871742b2b5972dd38aff1091, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682193229927 2023-04-22 19:53:53,551 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 0fd5c38ebb91470e8b5f93d817a32e81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1682193230631 2023-04-22 19:53:53,552 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 0a5263ac8ed74d938a199dfab7d38e5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682193231267 2023-04-22 19:53:53,552 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 2ef56d4f783346a29a9a4d8635e3c9a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682193231994 2023-04-22 19:53:53,553 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 58e7f21ea0f24898a219348f1eceac6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1682193232627 2023-04-22 19:53:53,553 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/c2febc5efbbb4f11a0f1a77d5c6a71e0 2023-04-22 19:53:53,557 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into b4abd2104a224688b157df112b83d600(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:53,557 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:53,557 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=11, startTime=1682193233456; duration=0sec 2023-04-22 19:53:53,557 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:53,569 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#C#compaction#80 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:53,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/726ce34271674d6399213fb53cd7168d 2023-04-22 19:53:53,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/c276f28f3efb4ffdad9419c0612cbbc3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/c276f28f3efb4ffdad9419c0612cbbc3 2023-04-22 19:53:53,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/c276f28f3efb4ffdad9419c0612cbbc3, entries=200, sequenceid=438, filesize=14.3 K 2023-04-22 19:53:53,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/c2febc5efbbb4f11a0f1a77d5c6a71e0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c2febc5efbbb4f11a0f1a77d5c6a71e0 2023-04-22 19:53:53,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c2febc5efbbb4f11a0f1a77d5c6a71e0, entries=150, sequenceid=438, filesize=11.9 K 2023-04-22 19:53:53,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/726ce34271674d6399213fb53cd7168d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/726ce34271674d6399213fb53cd7168d 2023-04-22 19:53:53,625 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:53,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:53,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/726ce34271674d6399213fb53cd7168d, entries=150, sequenceid=438, filesize=11.9 K 2023-04-22 19:53:53,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d6400a2b7b67145225eb12601c5e92eb in 150ms, sequenceid=438, compaction requested=false 2023-04-22 19:53:53,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:53,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:53,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:53,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 19:53:53,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:53,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:53,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:53,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:53,664 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:53,664 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:53,664 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:53,664 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:53,665 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,665 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,667 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,667 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:53,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:53,668 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,668 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:53,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:53,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:53,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:53,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:53,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:53,680 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:53,680 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:53,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:53,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:53,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,683 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,685 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,685 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:53,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e6a5851[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:53,686 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:53,687 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,688 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,688 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:53,688 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:53,688 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:53,689 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,689 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:53,690 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,690 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:53,690 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,690 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:53,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:53,696 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:53,696 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:53,696 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:53,696 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:53,696 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:53,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,701 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,704 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:53,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:53,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,710 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:53,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:53,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,711 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:53,711 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:53,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193293731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,811 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:53,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:53,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:53,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:53,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:53,815 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:53,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:53,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:53,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:53,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,816 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,816 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:53,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,817 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,817 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:53,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:53,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:53,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:53,818 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:53,818 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:53,818 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:53,819 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:53,819 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:53,819 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:53,819 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:53,821 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:53,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e70caf6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:53,823 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:53,823 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,824 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:53,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:53,826 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:53,826 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,826 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,826 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:53,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:53,829 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:53,829 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:53,829 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:53,829 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:53,829 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:53,830 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,830 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,830 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,832 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,833 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:53,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193293835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,837 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,837 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:53,837 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,837 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,837 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:53,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:53,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:53,837 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,837 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:53,837 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:53,837 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,837 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:53,837 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,837 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,937 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:53,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:53,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:53,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:53,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:53,941 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:53,941 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:53,941 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:53,941 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:53,942 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,942 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,943 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:53,943 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:53,944 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,944 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:53,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:53,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:53,945 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,945 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:53,945 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:53,945 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:53,947 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,947 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:53,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791069b1[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:53,949 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:53,949 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,950 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:53,951 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,951 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:53,954 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:53,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:53,957 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:53,957 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:53,957 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:53,958 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:53,958 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:53,959 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,959 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:53,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,959 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:53,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,960 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,961 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,964 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:53,964 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:53,964 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,964 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:53,964 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:53,964 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:53,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:53,964 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:53,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:53,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:53,964 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:53,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:53,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:53,965 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:53,965 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:53,997 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/1f978d9f8abe4a358049b9d082bdadc5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1f978d9f8abe4a358049b9d082bdadc5 2023-04-22 19:53:54,006 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/C of d6400a2b7b67145225eb12601c5e92eb into 1f978d9f8abe4a358049b9d082bdadc5(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:54,007 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:54,007 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/C, priority=11, startTime=1682193233456; duration=0sec 2023-04-22 19:53:54,007 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:54,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:54,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193294036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,065 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:54,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:54,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:54,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:54,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:54,072 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:54,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:54,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:54,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,075 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:54,075 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,076 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,076 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:54,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:54,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:54,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:54,080 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:54,080 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,080 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:54,083 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,083 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:54,083 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:54,083 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:54,086 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,086 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,086 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,086 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:54,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ac48eb[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:54,088 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:54,088 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,089 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,089 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:54,089 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:54,089 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,089 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:54,089 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:54,091 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:54,091 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,091 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,091 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:54,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:54,093 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:54,093 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:54,093 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:54,094 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:54,094 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:54,095 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,095 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,095 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:54,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,096 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,101 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:54,101 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:54,101 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:54,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:54,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:54,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:54,101 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:54,101 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,101 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,101 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/81f18a55af28462a897d9ad91e3a8fa9 2023-04-22 19:53:54,101 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,101 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:54,101 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:54,101 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:54,202 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:54,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:54,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:54,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:54,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:54,208 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:54,208 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,208 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:54,208 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:54,208 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,208 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,212 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:54,212 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:54,213 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,213 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:54,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:54,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:54,217 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:54,217 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:54,217 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,217 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:54,217 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:54,217 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,217 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:54,217 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:54,217 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,217 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:54,219 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,219 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:54,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e98363f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:54,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:54,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,224 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:54,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:54,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:54,224 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,224 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:54,226 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:54,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:54,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:54,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:54,232 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:54,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:54,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:54,233 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:54,233 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:54,234 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,234 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:54,234 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:54,235 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,237 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:54,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:54,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:54,243 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:54,243 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:54,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:54,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:54,243 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,244 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,244 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,244 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,244 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,244 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,244 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:54,244 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:54,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:54,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:54,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193294341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,344 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:54,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:54,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:54,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:54,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:54,347 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:54,347 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,347 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:54,348 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:54,348 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,348 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,350 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:54,350 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:54,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:54,351 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,351 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:54,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:54,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:54,352 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:54,352 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,352 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:54,352 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:54,352 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:54,352 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,352 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:54,352 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:54,352 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,352 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:54,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:54,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:54,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:54,354 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:54,355 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:54,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e77e60[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:54,357 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:54,357 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,358 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,358 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:54,358 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:54,358 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:54,359 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,359 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:54,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:54,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:54,362 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:54,363 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:54,363 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:54,363 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:54,363 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:54,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:54,366 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,368 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,372 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,372 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,372 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:54,372 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:54,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:54,373 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,373 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,373 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,373 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,373 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,373 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:54,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:54,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:54,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:54,373 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:54,373 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:54,474 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:54,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:54,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:54,478 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:54,478 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,478 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:54,478 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:54,479 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,479 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:54,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:54,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,481 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,481 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:54,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:54,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:54,483 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,483 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:54,483 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:54,483 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:54,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:54,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:54,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,485 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c40dcec[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:54,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:54,488 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,490 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:54,490 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,490 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:54,492 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:54,495 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:54,495 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:54,495 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:54,495 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:54,495 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:54,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:54,498 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,500 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,502 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,502 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:54,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:54,502 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,502 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:54,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:54,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:54,502 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,502 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:54,503 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,503 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,503 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,503 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,503 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:54,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:54,503 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:54,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:54,515 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/8d8944a9b56747eb841caa46d3326526 2023-04-22 19:53:54,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/f6be02e773f94a9aae5e2606e173f3b8 2023-04-22 19:53:54,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/81f18a55af28462a897d9ad91e3a8fa9 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/81f18a55af28462a897d9ad91e3a8fa9 2023-04-22 19:53:54,578 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/81f18a55af28462a897d9ad91e3a8fa9, entries=200, sequenceid=455, filesize=14.3 K 2023-04-22 19:53:54,584 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/8d8944a9b56747eb841caa46d3326526 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/8d8944a9b56747eb841caa46d3326526 2023-04-22 19:53:54,603 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:54,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:54,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:54,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/8d8944a9b56747eb841caa46d3326526, entries=150, sequenceid=455, filesize=11.9 K 2023-04-22 19:53:54,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:54,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:54,611 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:54,611 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,611 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:54,611 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:54,611 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,611 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/f6be02e773f94a9aae5e2606e173f3b8 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f6be02e773f94a9aae5e2606e173f3b8 2023-04-22 19:53:54,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:54,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:54,614 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,615 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,615 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:54,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:54,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:54,617 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,617 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:54,617 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:54,617 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:54,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cfb269f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:54,622 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:54,622 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,626 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,626 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f6be02e773f94a9aae5e2606e173f3b8, entries=150, sequenceid=455, filesize=11.9 K 2023-04-22 19:53:54,626 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,626 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:54,626 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:54,626 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:54,627 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:54,627 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d6400a2b7b67145225eb12601c5e92eb in 974ms, sequenceid=455, compaction requested=true 2023-04-22 19:53:54,628 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:54,628 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:54,628 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:54,628 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:54,628 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:54,628 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:54,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:54,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:54,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:54,629 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,629 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42290 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,630 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:54,630 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:53:54,630 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,630 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5e0ca88f7b3e4cc69aafaa52d2f0fbe1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/c276f28f3efb4ffdad9419c0612cbbc3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/81f18a55af28462a897d9ad91e3a8fa9] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=41.3 K 2023-04-22 19:53:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,630 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:53:54,630 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,630 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b4abd2104a224688b157df112b83d600, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c2febc5efbbb4f11a0f1a77d5c6a71e0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/8d8944a9b56747eb841caa46d3326526] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=36.5 K 2023-04-22 19:53:54,631 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 5e0ca88f7b3e4cc69aafaa52d2f0fbe1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1682193232627 2023-04-22 19:53:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,631 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c276f28f3efb4ffdad9419c0612cbbc3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682193233330 2023-04-22 19:53:54,632 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b4abd2104a224688b157df112b83d600, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1682193232627 2023-04-22 19:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,632 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 81f18a55af28462a897d9ad91e3a8fa9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682193233506 2023-04-22 19:53:54,632 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting c2febc5efbbb4f11a0f1a77d5c6a71e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682193233330 2023-04-22 19:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,633 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 8d8944a9b56747eb841caa46d3326526, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682193233530 2023-04-22 19:53:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:54,635 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:54,635 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:54,635 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:54,635 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:54,635 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:54,637 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,637 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:54,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,639 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,641 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,642 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:54,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:54,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:54,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:54,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:54,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:54,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:54,646 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:54,647 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:54,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:54,649 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#85 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:54,693 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/0da7fb150f5b4eb1855871fdb5566c24 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0da7fb150f5b4eb1855871fdb5566c24 2023-04-22 19:53:54,702 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into 0da7fb150f5b4eb1855871fdb5566c24(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:54,702 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:54,702 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=13, startTime=1682193234628; duration=0sec 2023-04-22 19:53:54,703 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:54,703 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:54,704 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:54,705 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/C is initiating minor compaction (all files) 2023-04-22 19:53:54,705 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/C in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,705 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1f978d9f8abe4a358049b9d082bdadc5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/726ce34271674d6399213fb53cd7168d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f6be02e773f94a9aae5e2606e173f3b8] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=36.5 K 2023-04-22 19:53:54,705 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1f978d9f8abe4a358049b9d082bdadc5, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1682193232627 2023-04-22 19:53:54,706 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 726ce34271674d6399213fb53cd7168d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682193233330 2023-04-22 19:53:54,706 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f6be02e773f94a9aae5e2606e173f3b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682193233530 2023-04-22 19:53:54,720 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#C#compaction#86 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:54,745 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/a1c0166a7118408b96074f8c90b826d7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/a1c0166a7118408b96074f8c90b826d7 2023-04-22 19:53:54,747 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:54,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:54,754 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/C of d6400a2b7b67145225eb12601c5e92eb into a1c0166a7118408b96074f8c90b826d7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:54,754 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:54,754 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/C, priority=13, startTime=1682193234628; duration=0sec 2023-04-22 19:53:54,754 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:54,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:54,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:54,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:54,760 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:54,760 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,760 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:54,760 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:54,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,762 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:54,762 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:54,763 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,763 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:54,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:54,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:54,777 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,777 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:54,777 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-22 19:53:54,777 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:54,777 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:54,777 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:54,777 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:54,777 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:54,778 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:54,793 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/dcfc1428b0304f91b2b330f5afb04fa2 2023-04-22 19:53:54,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:54,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:54,862 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/a973d36663484727a910341d990e11d9 2023-04-22 19:53:54,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:54,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193294890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,915 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/1aca6123e6c14a629282337d251707f3 2023-04-22 19:53:54,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/dcfc1428b0304f91b2b330f5afb04fa2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/dcfc1428b0304f91b2b330f5afb04fa2 2023-04-22 19:53:54,936 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/dcfc1428b0304f91b2b330f5afb04fa2, entries=150, sequenceid=475, filesize=11.9 K 2023-04-22 19:53:54,937 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/a973d36663484727a910341d990e11d9 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a973d36663484727a910341d990e11d9 2023-04-22 19:53:54,943 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a973d36663484727a910341d990e11d9, entries=150, sequenceid=475, filesize=11.9 K 2023-04-22 19:53:54,944 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/1aca6123e6c14a629282337d251707f3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1aca6123e6c14a629282337d251707f3 2023-04-22 19:53:54,977 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1aca6123e6c14a629282337d251707f3, entries=150, sequenceid=475, filesize=11.9 K 2023-04-22 19:53:54,978 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d6400a2b7b67145225eb12601c5e92eb in 201ms, sequenceid=475, compaction requested=false 2023-04-22 19:53:54,978 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:54,978 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:54,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:54,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:54,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,979 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:54,979 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:54,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:54,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33bdcaf1[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:54,989 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:54,990 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,991 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:54,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:54,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:54,998 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:54,998 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:54,998 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:54,998 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:54,998 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:54,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:54,998 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-22 19:53:54,998 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:54,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:54,999 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:54,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:54,999 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:54,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:55,000 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,000 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:55,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:55,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,006 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,011 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:55,011 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:55,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:55,011 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:55,011 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:55,012 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:55,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:55,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:55,012 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,012 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,012 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,012 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,012 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,012 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:55,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:55,012 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:55,035 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/3bbf81cd303049fdbc7d873a2ffb1e95 2023-04-22 19:53:55,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/b44c60f8f61949f9ac664e070123d23e 2023-04-22 19:53:55,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:55,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193295073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,089 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/b650032a513d45c0ae2e61e20d18fbdc as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/b650032a513d45c0ae2e61e20d18fbdc 2023-04-22 19:53:55,094 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into b650032a513d45c0ae2e61e20d18fbdc(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:55,094 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:55,094 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=13, startTime=1682193234628; duration=0sec 2023-04-22 19:53:55,094 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:55,112 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:55,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:55,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:55,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:55,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:55,116 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:55,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:55,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:55,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:55,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,117 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:55,117 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:55,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:55,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:55,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:55,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:55,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:55,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:55,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:55,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:55,120 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:55,120 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:55,120 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:55,120 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:55,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:55,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:55,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:55,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:55,121 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,121 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:55,122 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,122 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:55,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:55,123 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:55,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e64a698[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:55,123 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,124 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:55,125 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,125 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:55,127 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:55,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:55,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:55,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:55,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:55,129 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:55,129 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:55,129 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:55,129 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:55,130 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:55,131 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,131 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:55,131 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:55,132 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:55,134 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:55,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:55,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:55,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:55,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:55,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:55,137 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:55,137 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:55,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:55,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:55,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193295175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,237 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:55,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:55,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:55,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:55,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:55,241 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:55,241 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:55,241 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:55,241 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:55,241 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,242 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,243 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,243 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:55,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:55,243 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,243 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:55,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:55,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:55,245 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:55,245 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:55,245 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:55,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:55,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:55,248 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:55,248 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:55,248 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:55,248 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:55,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:55,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:55,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:55,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:55,251 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,251 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:55,251 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,252 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:55,253 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c9a204[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:55,253 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,255 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,255 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:55,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,255 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:55,255 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:55,255 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:55,259 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:55,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:55,262 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:55,262 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:55,262 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:55,262 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:55,262 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:55,264 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,265 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:55,265 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:55,265 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,266 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,269 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:55,269 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:55,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:55,269 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:55,269 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:55,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:55,269 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:55,269 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:55,269 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,269 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:55,269 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,269 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,269 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,270 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:55,370 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:55,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:55,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:55,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:55,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:55,373 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:55,373 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:55,373 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:55,374 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:55,374 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,374 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,375 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,375 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:55,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:55,375 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,375 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:55,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:55,377 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:55,377 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:55,377 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:55,377 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:55,377 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:55,377 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:55,377 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:55,377 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:55,377 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:55,377 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:55,377 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:55,377 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,377 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:55,377 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:55,379 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,379 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:55,379 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,379 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:55,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193295379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:55,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:55,381 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:55,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7183addc[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:55,381 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,388 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,388 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:55,388 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:55,388 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,389 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:55,391 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:55,394 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:55,394 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:55,394 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:55,394 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:55,394 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:55,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:55,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:55,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,396 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:55,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,398 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,406 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,406 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:55,406 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:55,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:55,406 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,407 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:55,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:55,407 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,407 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,407 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:55,407 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,407 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:55,407 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:55,407 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:55,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:55,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/fbe334ce4cee4c5484b6540847296219 2023-04-22 19:53:55,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/3bbf81cd303049fdbc7d873a2ffb1e95 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/3bbf81cd303049fdbc7d873a2ffb1e95 2023-04-22 19:53:55,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/3bbf81cd303049fdbc7d873a2ffb1e95, entries=200, sequenceid=491, filesize=14.3 K 2023-04-22 19:53:55,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/b44c60f8f61949f9ac664e070123d23e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b44c60f8f61949f9ac664e070123d23e 2023-04-22 19:53:55,507 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:55,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:55,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:55,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b44c60f8f61949f9ac664e070123d23e, entries=150, sequenceid=491, filesize=11.9 K 2023-04-22 19:53:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:55,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/fbe334ce4cee4c5484b6540847296219 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/fbe334ce4cee4c5484b6540847296219 2023-04-22 19:53:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:55,533 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:55,533 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:55,533 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:55,533 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:55,534 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,534 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:55,537 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:55,538 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,538 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:55,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:55,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:55,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:55,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/fbe334ce4cee4c5484b6540847296219, entries=150, sequenceid=491, filesize=11.9 K 2023-04-22 19:53:55,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for d6400a2b7b67145225eb12601c5e92eb in 543ms, sequenceid=491, compaction requested=true 2023-04-22 19:53:55,542 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:55,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:55,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:55,542 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:55,542 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:55,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:55,542 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:55,543 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:55,543 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-22 19:53:55,543 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:55,543 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:55,543 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:55,543 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:55,543 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:55,543 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:55,544 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39952 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:55,544 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:55,544 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:53:55,544 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:53:55,544 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:55,544 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:55,544 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/b650032a513d45c0ae2e61e20d18fbdc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/dcfc1428b0304f91b2b330f5afb04fa2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/3bbf81cd303049fdbc7d873a2ffb1e95] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=39.0 K 2023-04-22 19:53:55,544 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0da7fb150f5b4eb1855871fdb5566c24, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a973d36663484727a910341d990e11d9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b44c60f8f61949f9ac664e070123d23e] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=36.6 K 2023-04-22 19:53:55,545 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting b650032a513d45c0ae2e61e20d18fbdc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682193233530 2023-04-22 19:53:55,545 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0da7fb150f5b4eb1855871fdb5566c24, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682193233530 2023-04-22 19:53:55,545 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting dcfc1428b0304f91b2b330f5afb04fa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682193233720 2023-04-22 19:53:55,545 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a973d36663484727a910341d990e11d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682193233720 2023-04-22 19:53:55,546 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 3bbf81cd303049fdbc7d873a2ffb1e95, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1682193234851 2023-04-22 19:53:55,546 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b44c60f8f61949f9ac664e070123d23e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1682193234871 2023-04-22 19:53:55,559 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#93 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:55,578 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#95 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:55,633 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/0dcb3546aca042deacaa9bd5522c49dd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0dcb3546aca042deacaa9bd5522c49dd 2023-04-22 19:53:55,641 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into 0dcb3546aca042deacaa9bd5522c49dd(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:55,641 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:55,641 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=13, startTime=1682193235542; duration=0sec 2023-04-22 19:53:55,641 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:55,642 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:55,643 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:55,643 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/C is initiating minor compaction (all files) 2023-04-22 19:53:55,643 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/C in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:55,643 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/a1c0166a7118408b96074f8c90b826d7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1aca6123e6c14a629282337d251707f3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/fbe334ce4cee4c5484b6540847296219] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=36.6 K 2023-04-22 19:53:55,645 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a1c0166a7118408b96074f8c90b826d7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682193233530 2023-04-22 19:53:55,646 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1aca6123e6c14a629282337d251707f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682193233720 2023-04-22 19:53:55,648 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting fbe334ce4cee4c5484b6540847296219, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1682193234871 2023-04-22 19:53:55,669 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/198bdae3345143639d6e9c62d6753e2d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/198bdae3345143639d6e9c62d6753e2d 2023-04-22 19:53:55,674 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#C#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:55,675 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/a8615b96b7434801bd46c44a2b68241f 2023-04-22 19:53:55,677 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into 198bdae3345143639d6e9c62d6753e2d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:55,678 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:55,678 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=13, startTime=1682193235542; duration=0sec 2023-04-22 19:53:55,678 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:55,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:55,693 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:55,726 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/8478acaace4848b5b571949462fbf222 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/8478acaace4848b5b571949462fbf222 2023-04-22 19:53:55,735 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/C of d6400a2b7b67145225eb12601c5e92eb into 8478acaace4848b5b571949462fbf222(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:55,735 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:55,735 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/C, priority=13, startTime=1682193235542; duration=0sec 2023-04-22 19:53:55,735 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:55,739 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/0c4efd388ae94c6e9efb42c5b7634580 2023-04-22 19:53:55,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:55,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193295752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:55,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53790 deadline: 1682193295753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,759 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:53:55,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:55,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53784 deadline: 1682193295770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,777 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:53:55,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1682193295778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:55,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53802 deadline: 1682193295779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:55,792 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:53:55,792 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:53:55,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:55,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193295855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:56,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193296059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,197 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/7a205c37f8534dc7a05e34ac13b6dc53 2023-04-22 19:53:56,203 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/a8615b96b7434801bd46c44a2b68241f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a8615b96b7434801bd46c44a2b68241f 2023-04-22 19:53:56,208 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a8615b96b7434801bd46c44a2b68241f, entries=150, sequenceid=508, filesize=11.9 K 2023-04-22 19:53:56,209 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/0c4efd388ae94c6e9efb42c5b7634580 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0c4efd388ae94c6e9efb42c5b7634580 2023-04-22 19:53:56,215 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0c4efd388ae94c6e9efb42c5b7634580, entries=150, sequenceid=508, filesize=11.9 K 2023-04-22 19:53:56,216 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/7a205c37f8534dc7a05e34ac13b6dc53 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7a205c37f8534dc7a05e34ac13b6dc53 2023-04-22 19:53:56,222 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7a205c37f8534dc7a05e34ac13b6dc53, entries=150, sequenceid=508, filesize=11.9 K 2023-04-22 19:53:56,223 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for d6400a2b7b67145225eb12601c5e92eb in 680ms, sequenceid=508, compaction requested=false 2023-04-22 19:53:56,223 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:56,223 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:56,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:56,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:56,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:56,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,229 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:56,229 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,229 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:56,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:56,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cb40b5f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:56,231 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:56,231 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,233 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,233 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:56,233 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,233 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:56,233 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:56,233 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:56,234 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:56,234 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,234 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:56,234 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:56,237 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:56,237 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:56,238 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:56,238 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:56,238 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:56,240 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,240 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,240 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:56,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,242 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,244 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:56,247 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:56,248 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:56,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:56,248 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,248 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,248 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:56,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:56,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:56,248 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:56,248 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:56,248 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:56,248 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,248 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,248 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,248 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:56,348 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:56,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:56,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:56,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:56,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:56,352 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:56,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:56,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:56,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:56,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,354 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:56,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:56,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:56,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:56,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:56,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:56,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:56,356 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-22 19:53:56,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:56,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:56,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:56,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:56,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:56,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:56,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:56,365 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:56,366 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/9f4a0acda2cd48178c2e56541330f4ca 2023-04-22 19:53:56,398 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/e0953d2637614f728cda618239cf2aa5 2023-04-22 19:53:56,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:56,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193296417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,446 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/489919b6cc9a417ca9d708e818607dd4 2023-04-22 19:53:56,453 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/9f4a0acda2cd48178c2e56541330f4ca as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9f4a0acda2cd48178c2e56541330f4ca 2023-04-22 19:53:56,459 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9f4a0acda2cd48178c2e56541330f4ca, entries=150, sequenceid=527, filesize=11.9 K 2023-04-22 19:53:56,460 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/e0953d2637614f728cda618239cf2aa5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e0953d2637614f728cda618239cf2aa5 2023-04-22 19:53:56,466 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e0953d2637614f728cda618239cf2aa5, entries=150, sequenceid=527, filesize=11.9 K 2023-04-22 19:53:56,467 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/489919b6cc9a417ca9d708e818607dd4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/489919b6cc9a417ca9d708e818607dd4 2023-04-22 19:53:56,471 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/489919b6cc9a417ca9d708e818607dd4, entries=150, sequenceid=527, filesize=11.9 K 2023-04-22 19:53:56,472 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for d6400a2b7b67145225eb12601c5e92eb in 116ms, sequenceid=527, compaction requested=true 2023-04-22 19:53:56,473 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:56,473 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:56,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:56,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:56,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:56,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:56,474 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,474 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:56,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:56,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773f2837[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:56,479 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:56,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,482 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:56,483 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,483 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:56,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:56,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:56,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:56,488 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:56,488 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:56,488 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:56,488 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:56,488 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:56,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:56,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:56,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:56,492 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,493 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,501 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,501 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:56,501 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:56,501 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,501 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:56,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:56,504 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,504 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,504 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,504 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:56,504 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:56,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:56,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:56,504 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:56,504 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:56,528 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-22 19:53:56,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:56,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:56,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:56,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:56,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:56,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:56,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:56,605 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:56,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:56,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:56,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:56,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:56,616 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:56,616 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:56,616 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:56,616 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:56,617 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,617 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:56,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193296618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:56,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,620 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,621 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,621 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:56,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:56,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:56,624 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:56,624 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:56,624 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:56,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:56,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:56,629 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:56,629 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:56,629 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:56,629 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:56,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:56,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:56,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:56,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:56,631 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:56,632 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:56,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:56,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e5f7d56[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:56,634 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:56,634 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:56,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:56,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:56,638 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,638 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:56,640 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:56,640 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,640 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:56,640 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:56,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:56,643 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:56,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:56,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:56,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:56,643 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:56,647 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,647 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:56,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:56,648 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:56,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:56,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:56,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:56,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:56,657 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:56,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:56,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:56,657 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:56,657 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:56,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193296720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,757 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:56,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:56,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:56,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:56,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:56,764 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:56,764 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:56,764 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:56,764 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:56,765 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,765 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,821 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:56,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:56,821 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,822 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,822 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:56,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:56,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:56,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:56,839 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:56,839 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:56,839 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:56,842 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:56,843 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:56,843 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:56,843 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:56,843 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:56,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,891 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:56,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:56,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:56,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:56,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:56,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22dec40c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:56,894 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:56,894 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:56,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193296926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,967 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:56,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:56,968 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:56,968 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:56,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:56,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:56,998 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/60c2e58d677247d9b3d4ddf99b8d640e 2023-04-22 19:53:57,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/a1957feecd024b149a3102820ba0b32c 2023-04-22 19:53:57,021 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:57,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:57,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:57,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/c605e0bb174749968fd3735beffb4276 2023-04-22 19:53:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:57,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:57,025 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:57,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:57,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:57,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:57,025 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:57,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/60c2e58d677247d9b3d4ddf99b8d640e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/60c2e58d677247d9b3d4ddf99b8d640e 2023-04-22 19:53:57,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/60c2e58d677247d9b3d4ddf99b8d640e, entries=150, sequenceid=544, filesize=11.9 K 2023-04-22 19:53:57,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/a1957feecd024b149a3102820ba0b32c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a1957feecd024b149a3102820ba0b32c 2023-04-22 19:53:57,040 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a1957feecd024b149a3102820ba0b32c, entries=150, sequenceid=544, filesize=11.9 K 2023-04-22 19:53:57,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/c605e0bb174749968fd3735beffb4276 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c605e0bb174749968fd3735beffb4276 2023-04-22 19:53:57,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c605e0bb174749968fd3735beffb4276, entries=150, sequenceid=544, filesize=11.9 K 2023-04-22 19:53:57,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d6400a2b7b67145225eb12601c5e92eb in 517ms, sequenceid=544, compaction requested=true 2023-04-22 19:53:57,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:57,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:57,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:57,046 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:57,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:57,046 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:57,048 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:57,048 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:57,048 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:53:57,048 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:53:57,048 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:57,048 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:57,048 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/198bdae3345143639d6e9c62d6753e2d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a8615b96b7434801bd46c44a2b68241f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9f4a0acda2cd48178c2e56541330f4ca, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/60c2e58d677247d9b3d4ddf99b8d640e] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=48.6 K 2023-04-22 19:53:57,048 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0dcb3546aca042deacaa9bd5522c49dd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0c4efd388ae94c6e9efb42c5b7634580, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e0953d2637614f728cda618239cf2aa5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a1957feecd024b149a3102820ba0b32c] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=48.6 K 2023-04-22 19:53:57,049 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 198bdae3345143639d6e9c62d6753e2d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1682193234871 2023-04-22 19:53:57,049 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0dcb3546aca042deacaa9bd5522c49dd, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1682193234871 2023-04-22 19:53:57,049 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting a8615b96b7434801bd46c44a2b68241f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1682193235042 2023-04-22 19:53:57,049 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0c4efd388ae94c6e9efb42c5b7634580, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1682193235042 2023-04-22 19:53:57,049 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 9f4a0acda2cd48178c2e56541330f4ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1682193235717 2023-04-22 19:53:57,050 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting e0953d2637614f728cda618239cf2aa5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1682193235717 2023-04-22 19:53:57,050 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 60c2e58d677247d9b3d4ddf99b8d640e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1682193236393 2023-04-22 19:53:57,050 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a1957feecd024b149a3102820ba0b32c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1682193236393 2023-04-22 19:53:57,059 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#106 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:57,060 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#105 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:57,075 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,075 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:57,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,075 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:57,076 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:57,077 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:57,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:57,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:57,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:57,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:57,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:57,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:57,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:57,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:57,176 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:57,176 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:57,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:57,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:57,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:57,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:57,231 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-22 19:53:57,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:57,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:57,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:57,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:57,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:57,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:57,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/7e34c891a86c49c8b942f9cccca25ad6 2023-04-22 19:53:57,276 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:57,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:57,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:57,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:57,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:57,288 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:57,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:57,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:57,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:57,289 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,289 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:57,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:57,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193297287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/7c923c40217e4298b4a614187d730767 2023-04-22 19:53:57,325 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/1dc7525130e84f51ae205ce5fbaf67c7 2023-04-22 19:53:57,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/7e34c891a86c49c8b942f9cccca25ad6 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e34c891a86c49c8b942f9cccca25ad6 2023-04-22 19:53:57,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e34c891a86c49c8b942f9cccca25ad6, entries=200, sequenceid=560, filesize=14.3 K 2023-04-22 19:53:57,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/7c923c40217e4298b4a614187d730767 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7c923c40217e4298b4a614187d730767 2023-04-22 19:53:57,344 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:57,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:57,345 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,345 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:57,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:57,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:57,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:57,346 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:57,346 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:57,346 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:57,346 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:57,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7c923c40217e4298b4a614187d730767, entries=150, sequenceid=560, filesize=11.9 K 2023-04-22 19:53:57,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/1dc7525130e84f51ae205ce5fbaf67c7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1dc7525130e84f51ae205ce5fbaf67c7 2023-04-22 19:53:57,356 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1dc7525130e84f51ae205ce5fbaf67c7, entries=150, sequenceid=560, filesize=11.9 K 2023-04-22 19:53:57,357 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for d6400a2b7b67145225eb12601c5e92eb in 126ms, sequenceid=560, compaction requested=true 2023-04-22 19:53:57,357 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:57,357 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-22 19:53:57,357 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-22 19:53:57,357 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-22 19:53:57,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:57,395 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-22 19:53:57,395 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:57,395 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:57,395 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:57,395 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:57,395 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:57,395 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:57,421 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:57,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:57,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:57,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e3ccdfb[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:57,423 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:57,423 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:57,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:57,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193297430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,490 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:57,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:57,491 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:57,491 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:57,491 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:57,491 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,491 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:57,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 424 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193297533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:57,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:57,540 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:57,540 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:57,540 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:57,540 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:57,540 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:57,562 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/5c5fc24b23d949d69a5b8352b05d9696 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5c5fc24b23d949d69a5b8352b05d9696 2023-04-22 19:53:57,564 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/c84d9f72547d49908543790a85606c9a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c84d9f72547d49908543790a85606c9a 2023-04-22 19:53:57,573 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into 5c5fc24b23d949d69a5b8352b05d9696(size=13.1 K), total size for store is 27.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:57,573 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into c84d9f72547d49908543790a85606c9a(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:57,573 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:57,573 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:57,574 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=12, startTime=1682193237046; duration=0sec 2023-04-22 19:53:57,574 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=12, startTime=1682193237046; duration=0sec 2023-04-22 19:53:57,574 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-22 19:53:57,574 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-22 19:53:57,574 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:53:57,575 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,575 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:57,575 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:57,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:57,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:57,576 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:57,576 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61986 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:53:57,577 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/C is initiating minor compaction (all files) 2023-04-22 19:53:57,577 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/C in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:57,577 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 5 compacting, 0 eligible, 16 blocking 2023-04-22 19:53:57,577 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 19:53:57,577 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 19:53:57,577 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. because compaction request was cancelled 2023-04-22 19:53:57,577 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/8478acaace4848b5b571949462fbf222, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7a205c37f8534dc7a05e34ac13b6dc53, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/489919b6cc9a417ca9d708e818607dd4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c605e0bb174749968fd3735beffb4276, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1dc7525130e84f51ae205ce5fbaf67c7] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=60.5 K 2023-04-22 19:53:57,577 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-22 19:53:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:57,578 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 8478acaace4848b5b571949462fbf222, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1682193234871 2023-04-22 19:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,578 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 19:53:57,578 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 19:53:57,578 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. because compaction request was cancelled 2023-04-22 19:53:57,578 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 7a205c37f8534dc7a05e34ac13b6dc53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1682193235042 2023-04-22 19:53:57,578 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-22 19:53:57,579 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:57,579 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 489919b6cc9a417ca9d708e818607dd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1682193235717 2023-04-22 19:53:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:57,579 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c605e0bb174749968fd3735beffb4276, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1682193236393 2023-04-22 19:53:57,580 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 19:53:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,580 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 19:53:57,580 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. because compaction request was cancelled 2023-04-22 19:53:57,580 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 1dc7525130e84f51ae205ce5fbaf67c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1682193236608 2023-04-22 19:53:57,594 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#C#compaction#111 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:57,626 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/de76b7ef4db740ab92faa35affebd069 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/de76b7ef4db740ab92faa35affebd069 2023-04-22 19:53:57,633 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/C of d6400a2b7b67145225eb12601c5e92eb into de76b7ef4db740ab92faa35affebd069(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:57,633 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:57,633 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/C, priority=11, startTime=1682193237357; duration=0sec 2023-04-22 19:53:57,634 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:57,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:57,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193297739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,775 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:57,775 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:57,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:57,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:57,775 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:57,775 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:57,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:57,775 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:57,775 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:57,776 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:57,775 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:57,776 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:57,776 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:57,776 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:57,804 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/7e666c8737854fc2a16d868f7bf9ba28 2023-04-22 19:53:57,817 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/7e1065fd155d4abca116bc4ca501e0d7 2023-04-22 19:53:57,876 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:57,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:57,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:57,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:57,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:57,879 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:57,879 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:57,879 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:57,879 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:57,880 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,880 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:57,967 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:57,967 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:57,968 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,968 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:57,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:57,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:57,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:57,970 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:57,970 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:57,970 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:57,970 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:58,036 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,036 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:58,037 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,037 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:58,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:58,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20cc369b[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:58,039 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:58,039 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:58,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193298040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,121 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:58,121 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,121 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:58,167 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:58,170 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:58,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:58,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:58,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:58,170 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:58,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:58,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:58,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:58,230 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,231 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,245 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/3b681b08bf7a4e0daa067b4be01003f8 2023-04-22 19:53:58,251 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/7e666c8737854fc2a16d868f7bf9ba28 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e666c8737854fc2a16d868f7bf9ba28 2023-04-22 19:53:58,253 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,253 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:58,253 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:58,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:58,253 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:58,253 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:58,253 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:53:58,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:58,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:58,253 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:58,253 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,254 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,254 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:58,254 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:58,254 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:58,255 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e666c8737854fc2a16d868f7bf9ba28, entries=200, sequenceid=577, filesize=14.3 K 2023-04-22 19:53:58,256 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/7e1065fd155d4abca116bc4ca501e0d7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7e1065fd155d4abca116bc4ca501e0d7 2023-04-22 19:53:58,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7e1065fd155d4abca116bc4ca501e0d7, entries=150, sequenceid=577, filesize=11.9 K 2023-04-22 19:53:58,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/3b681b08bf7a4e0daa067b4be01003f8 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3b681b08bf7a4e0daa067b4be01003f8 2023-04-22 19:53:58,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3b681b08bf7a4e0daa067b4be01003f8, entries=150, sequenceid=577, filesize=11.9 K 2023-04-22 19:53:58,269 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d6400a2b7b67145225eb12601c5e92eb in 874ms, sequenceid=577, compaction requested=true 2023-04-22 19:53:58,269 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:58,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:58,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:58,269 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:58,269 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:58,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:58,272 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42630 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:58,272 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:58,272 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:53:58,272 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:53:58,272 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:58,272 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:58,272 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5c5fc24b23d949d69a5b8352b05d9696, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e34c891a86c49c8b942f9cccca25ad6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e666c8737854fc2a16d868f7bf9ba28] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=41.6 K 2023-04-22 19:53:58,272 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c84d9f72547d49908543790a85606c9a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7c923c40217e4298b4a614187d730767, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7e1065fd155d4abca116bc4ca501e0d7] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=36.9 K 2023-04-22 19:53:58,273 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 5c5fc24b23d949d69a5b8352b05d9696, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1682193236393 2023-04-22 19:53:58,273 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting c84d9f72547d49908543790a85606c9a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1682193236393 2023-04-22 19:53:58,273 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 7e34c891a86c49c8b942f9cccca25ad6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1682193236605 2023-04-22 19:53:58,273 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 7c923c40217e4298b4a614187d730767, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1682193236608 2023-04-22 19:53:58,274 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 7e666c8737854fc2a16d868f7bf9ba28, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1682193237261 2023-04-22 19:53:58,274 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 7e1065fd155d4abca116bc4ca501e0d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1682193237277 2023-04-22 19:53:58,282 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:58,283 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#115 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:58,296 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/45783de6a9564298ba7ef564c3d3a1f1 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/45783de6a9564298ba7ef564c3d3a1f1 2023-04-22 19:53:58,302 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into 45783de6a9564298ba7ef564c3d3a1f1(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:58,302 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:58,302 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=13, startTime=1682193238269; duration=0sec 2023-04-22 19:53:58,302 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:58,303 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-22 19:53:58,303 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 19:53:58,303 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 19:53:58,304 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. because compaction request was cancelled 2023-04-22 19:53:58,354 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:58,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:58,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:58,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:58,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:58,359 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:58,359 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:58,359 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:58,359 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:58,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:58,413 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,413 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:58,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:58,414 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,414 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:58,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:58,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:58,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:58,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:58,416 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:58,416 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:58,416 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 19:53:58,416 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:58,417 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:58,417 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:58,417 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:58,417 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:58,417 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:58,426 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=596 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/01a6239f38e24158bc6f6920ea4fb312 2023-04-22 19:53:58,440 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=596 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/691be9325f2b498ca90ddd20f4c626da 2023-04-22 19:53:58,463 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=596 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/56b271f6a1e4453a8da4ed4513a18b1d 2023-04-22 19:53:58,469 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/01a6239f38e24158bc6f6920ea4fb312 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/01a6239f38e24158bc6f6920ea4fb312 2023-04-22 19:53:58,475 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/01a6239f38e24158bc6f6920ea4fb312, entries=150, sequenceid=596, filesize=11.9 K 2023-04-22 19:53:58,477 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/691be9325f2b498ca90ddd20f4c626da as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/691be9325f2b498ca90ddd20f4c626da 2023-04-22 19:53:58,487 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/691be9325f2b498ca90ddd20f4c626da, entries=150, sequenceid=596, filesize=11.9 K 2023-04-22 19:53:58,488 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/56b271f6a1e4453a8da4ed4513a18b1d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/56b271f6a1e4453a8da4ed4513a18b1d 2023-04-22 19:53:58,496 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/56b271f6a1e4453a8da4ed4513a18b1d, entries=150, sequenceid=596, filesize=11.9 K 2023-04-22 19:53:58,497 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for d6400a2b7b67145225eb12601c5e92eb in 81ms, sequenceid=596, compaction requested=true 2023-04-22 19:53:58,497 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:58,497 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:58,498 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:58,498 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:58,498 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,499 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:58,499 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:58,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,529 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:58,529 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,530 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@154c4a19[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:58,531 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:58,532 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,559 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:58,560 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,560 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:58,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:58,565 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:53:58,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:58,565 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:58,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:58,565 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:58,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:58,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:53:58,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/a6100e48691647759db7133d071068bf 2023-04-22 19:53:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:58,637 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:58,637 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:58,637 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:58,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193238612, max timestamp=1682193238638Num uniques 859; 2023-04-22 19:53:58,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:53:58,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193238612, max timestamp=1682193238638Num uniques 859; 2023-04-22 19:53:58,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193238612, max timestamp=1682193238638Num uniques 859; 2023-04-22 19:53:58,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:53:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:58,642 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:58,642 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:58,642 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:58,642 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:58,642 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:58,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:53:58,694 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/79783a9a97204b0d850c2b3e2a10e612 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/79783a9a97204b0d850c2b3e2a10e612 2023-04-22 19:53:58,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:58,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,700 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into 79783a9a97204b0d850c2b3e2a10e612(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:58,700 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:58,700 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=13, startTime=1682193238269; duration=0sec 2023-04-22 19:53:58,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:58,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 470 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193298696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,700 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:58,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,705 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,706 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:58,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:58,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,767 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:58,767 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:58,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:58,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:58,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:58,768 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:58,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:58,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:58,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:58,768 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:58,768 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:58,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:58,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:58,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 472 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193298802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,868 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:58,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:58,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:58,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:58,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:58,873 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:58,873 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:58,873 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:58,873 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:58,873 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,873 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:58,944 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,944 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:58,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:58,945 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,945 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:58,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:58,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:58,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:58,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:58,946 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:58,946 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:58,946 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:58,946 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:53:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:53:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:53:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:53:59,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:59,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 474 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193299004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/c3c64310097c4a43b787382451aba65a 2023-04-22 19:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:59,013 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:59,014 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:59,014 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41e61218[Count = 0] remaining members to acquire global barrier 2023-04-22 19:53:59,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:53:59,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:59,060 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:59,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:59,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:53:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:53:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:53:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:53:59,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,137 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:53:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:59,137 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:59,137 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:53:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:53:59,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:53:59,140 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:53:59,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:53:59,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:53:59,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:53:59,140 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:53:59,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:59,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:53:59,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:59,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:53:59,176 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:53:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:59,177 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:59,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:53:59,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:59,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:53:59,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:53:59,229 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:53:59,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:59,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:59,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:59,229 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:53:59,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:53:59,229 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:53:59,230 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:59,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:53:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:53:59,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:59,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193299308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,330 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:53:59,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:53:59,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:53:59,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:53:59,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:53:59,333 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:53:59,333 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:59,333 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:53:59,333 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:53:59,334 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:59,334 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:59,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/f75cc290bb7f4eb4a8997716362f5f63 2023-04-22 19:53:59,428 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/a6100e48691647759db7133d071068bf as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a6100e48691647759db7133d071068bf 2023-04-22 19:53:59,434 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a6100e48691647759db7133d071068bf, entries=200, sequenceid=606, filesize=14.3 K 2023-04-22 19:53:59,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/c3c64310097c4a43b787382451aba65a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c3c64310097c4a43b787382451aba65a 2023-04-22 19:53:59,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c3c64310097c4a43b787382451aba65a, entries=150, sequenceid=606, filesize=11.9 K 2023-04-22 19:53:59,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/f75cc290bb7f4eb4a8997716362f5f63 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f75cc290bb7f4eb4a8997716362f5f63 2023-04-22 19:53:59,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f75cc290bb7f4eb4a8997716362f5f63, entries=150, sequenceid=606, filesize=11.9 K 2023-04-22 19:53:59,448 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d6400a2b7b67145225eb12601c5e92eb in 883ms, sequenceid=606, compaction requested=true 2023-04-22 19:53:59,449 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:59,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:59,449 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:59,449 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:53:59,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:59,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:59,450 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:59,450 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40292 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:53:59,450 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:53:59,450 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:53:59,450 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:59,450 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:59,450 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/45783de6a9564298ba7ef564c3d3a1f1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/691be9325f2b498ca90ddd20f4c626da, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c3c64310097c4a43b787382451aba65a] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=37.0 K 2023-04-22 19:53:59,450 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/79783a9a97204b0d850c2b3e2a10e612, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/01a6239f38e24158bc6f6920ea4fb312, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a6100e48691647759db7133d071068bf] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=39.3 K 2023-04-22 19:53:59,451 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 45783de6a9564298ba7ef564c3d3a1f1, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1682193237277 2023-04-22 19:53:59,451 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 79783a9a97204b0d850c2b3e2a10e612, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1682193237277 2023-04-22 19:53:59,451 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 691be9325f2b498ca90ddd20f4c626da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=596, earliestPutTs=1682193237413 2023-04-22 19:53:59,451 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 01a6239f38e24158bc6f6920ea4fb312, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=596, earliestPutTs=1682193237413 2023-04-22 19:53:59,452 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting c3c64310097c4a43b787382451aba65a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1682193238559 2023-04-22 19:53:59,452 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting a6100e48691647759db7133d071068bf, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1682193238559 2023-04-22 19:53:59,459 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#122 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:59,461 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#123 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:59,473 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/1382fd68715445ef94ef11babc69c4a3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1382fd68715445ef94ef11babc69c4a3 2023-04-22 19:53:59,478 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into 1382fd68715445ef94ef11babc69c4a3(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:59,478 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:59,478 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=13, startTime=1682193239449; duration=0sec 2023-04-22 19:53:59,478 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:53:59,478 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:53:59,481 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49970 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:53:59,481 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/C is initiating minor compaction (all files) 2023-04-22 19:53:59,481 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/C in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:59,481 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/de76b7ef4db740ab92faa35affebd069, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3b681b08bf7a4e0daa067b4be01003f8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/56b271f6a1e4453a8da4ed4513a18b1d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f75cc290bb7f4eb4a8997716362f5f63] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=48.8 K 2023-04-22 19:53:59,481 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting de76b7ef4db740ab92faa35affebd069, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1682193236608 2023-04-22 19:53:59,482 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 3b681b08bf7a4e0daa067b4be01003f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1682193237277 2023-04-22 19:53:59,483 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 56b271f6a1e4453a8da4ed4513a18b1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=596, earliestPutTs=1682193237413 2023-04-22 19:53:59,484 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting f75cc290bb7f4eb4a8997716362f5f63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1682193238559 2023-04-22 19:53:59,492 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#C#compaction#124 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:53:59,536 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:53:59,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:53:59,537 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,537 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:59,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:53:59,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:53:59,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:53:59,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:53:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:53:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:53:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:53:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:53:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:53:59,539 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:53:59,539 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. started... 2023-04-22 19:53:59,539 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-22 19:53:59,540 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:53:59,540 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:53:59,540 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:53:59,540 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:53:59,540 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:53:59,540 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:53:59,552 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/273ad6fd24d94584a81572bd62782f0e 2023-04-22 19:53:59,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:53:59,813 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. as already flushing 2023-04-22 19:53:59,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:59,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 485 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193299833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,872 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/0668bea639104f5389ef7fba0e83bd40 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0668bea639104f5389ef7fba0e83bd40 2023-04-22 19:53:59,878 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into 0668bea639104f5389ef7fba0e83bd40(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:59,878 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:59,878 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=13, startTime=1682193239449; duration=0sec 2023-04-22 19:53:59,878 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:59,903 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/26f7d76cf93f485597d9838375dc6a63 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/26f7d76cf93f485597d9838375dc6a63 2023-04-22 19:53:59,912 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/C of d6400a2b7b67145225eb12601c5e92eb into 26f7d76cf93f485597d9838375dc6a63(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:53:59,912 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:53:59,912 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/C, priority=12, startTime=1682193239449; duration=0sec 2023-04-22 19:53:59,912 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:53:59,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:53:59,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 487 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193299935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:53:59,971 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/26f32f18372648e9be0976ed86d52eaa 2023-04-22 19:54:00,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:00,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 489 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53810 deadline: 1682193300141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,385 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/10d890511baf4680b035a197a2cb7cae 2023-04-22 19:54:00,392 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/273ad6fd24d94584a81572bd62782f0e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/273ad6fd24d94584a81572bd62782f0e 2023-04-22 19:54:00,397 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/273ad6fd24d94584a81572bd62782f0e, entries=150, sequenceid=644, filesize=11.9 K 2023-04-22 19:54:00,398 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/26f32f18372648e9be0976ed86d52eaa as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/26f32f18372648e9be0976ed86d52eaa 2023-04-22 19:54:00,403 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/26f32f18372648e9be0976ed86d52eaa, entries=150, sequenceid=644, filesize=11.9 K 2023-04-22 19:54:00,404 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/10d890511baf4680b035a197a2cb7cae as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/10d890511baf4680b035a197a2cb7cae 2023-04-22 19:54:00,411 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/10d890511baf4680b035a197a2cb7cae, entries=150, sequenceid=644, filesize=11.9 K 2023-04-22 19:54:00,412 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for d6400a2b7b67145225eb12601c5e92eb in 873ms, sequenceid=644, compaction requested=false 2023-04-22 19:54:00,412 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:54:00,412 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:54:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:00,414 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,414 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:00,415 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:00,415 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9fd5cca[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:00,418 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:00,418 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:00,419 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:00,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:00,419 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:00,419 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:00,419 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:00,420 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,420 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:00,421 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:00,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:00,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:00,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:00,424 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:00,424 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:00,424 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:00,424 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:00,424 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:00,427 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:00,427 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:00,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:00,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:00,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:00,429 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,430 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,433 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,433 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:00,433 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:00,433 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:00,433 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:00,433 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:00,433 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:00,433 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:00,433 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:00,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:00,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:00,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:00,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:00,433 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:00,433 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:00,434 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:00,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:00,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:54:00,447 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 19:54:00,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:54:00,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:00,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:54:00,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:00,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:54:00,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:00,499 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49a45b6d to 127.0.0.1:53381 2023-04-22 19:54:00,499 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:00,500 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fea27db to 127.0.0.1:53381 2023-04-22 19:54:00,500 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:00,505 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0dbb9ac1 to 127.0.0.1:53381 2023-04-22 19:54:00,505 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:00,506 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:53381 2023-04-22 19:54:00,506 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:00,506 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd3abbd to 127.0.0.1:53381 2023-04-22 19:54:00,506 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:00,534 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:00,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:00,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/909c96ab85904150a3177df753552bec 2023-04-22 19:54:01,295 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/47fd7720795045859d4af0d0620c6ced 2023-04-22 19:54:01,706 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/2154edfcb4b0405bbb755b8366af602a 2023-04-22 19:54:01,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/909c96ab85904150a3177df753552bec as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/909c96ab85904150a3177df753552bec 2023-04-22 19:54:01,716 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/909c96ab85904150a3177df753552bec, entries=150, sequenceid=657, filesize=11.9 K 2023-04-22 19:54:01,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/47fd7720795045859d4af0d0620c6ced as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/47fd7720795045859d4af0d0620c6ced 2023-04-22 19:54:01,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/47fd7720795045859d4af0d0620c6ced, entries=100, sequenceid=657, filesize=9.5 K 2023-04-22 19:54:01,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/2154edfcb4b0405bbb755b8366af602a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2154edfcb4b0405bbb755b8366af602a 2023-04-22 19:54:01,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2154edfcb4b0405bbb755b8366af602a, entries=100, sequenceid=657, filesize=9.5 K 2023-04-22 19:54:01,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=114.05 KB/116790 for d6400a2b7b67145225eb12601c5e92eb in 1280ms, sequenceid=657, compaction requested=true 2023-04-22 19:54:01,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:54:01,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:01,727 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:01,727 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:01,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:01,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:01,728 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37950 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:01,728 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/A is initiating minor compaction (all files) 2023-04-22 19:54:01,729 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:01,729 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/A in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:54:01,729 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/B is initiating minor compaction (all files) 2023-04-22 19:54:01,729 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1382fd68715445ef94ef11babc69c4a3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/273ad6fd24d94584a81572bd62782f0e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/909c96ab85904150a3177df753552bec] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=37.1 K 2023-04-22 19:54:01,729 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/B in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:54:01,729 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0668bea639104f5389ef7fba0e83bd40, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/26f32f18372648e9be0976ed86d52eaa, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/47fd7720795045859d4af0d0620c6ced] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=34.7 K 2023-04-22 19:54:01,729 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 1382fd68715445ef94ef11babc69c4a3, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1682193238559 2023-04-22 19:54:01,729 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0668bea639104f5389ef7fba0e83bd40, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1682193238559 2023-04-22 19:54:01,730 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 273ad6fd24d94584a81572bd62782f0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1682193238688 2023-04-22 19:54:01,730 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 26f32f18372648e9be0976ed86d52eaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1682193238688 2023-04-22 19:54:01,730 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 909c96ab85904150a3177df753552bec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1682193239829 2023-04-22 19:54:01,730 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 47fd7720795045859d4af0d0620c6ced, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1682193239832 2023-04-22 19:54:01,737 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#A#compaction#131 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:01,738 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#B#compaction#132 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:01,752 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/9e18195bc38748aca4e0b21c5b013145 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/9e18195bc38748aca4e0b21c5b013145 2023-04-22 19:54:01,757 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/B of d6400a2b7b67145225eb12601c5e92eb into 9e18195bc38748aca4e0b21c5b013145(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:01,757 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:54:01,757 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/B, priority=13, startTime=1682193241727; duration=0sec 2023-04-22 19:54:01,757 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:01,757 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:01,758 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35476 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:01,759 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): d6400a2b7b67145225eb12601c5e92eb/C is initiating minor compaction (all files) 2023-04-22 19:54:01,759 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6400a2b7b67145225eb12601c5e92eb/C in TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:54:01,759 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/26f7d76cf93f485597d9838375dc6a63, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/10d890511baf4680b035a197a2cb7cae, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2154edfcb4b0405bbb755b8366af602a] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp, totalSize=34.6 K 2023-04-22 19:54:01,759 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 26f7d76cf93f485597d9838375dc6a63, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1682193238559 2023-04-22 19:54:01,760 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 10d890511baf4680b035a197a2cb7cae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1682193238688 2023-04-22 19:54:01,760 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 2154edfcb4b0405bbb755b8366af602a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1682193239832 2023-04-22 19:54:01,768 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): d6400a2b7b67145225eb12601c5e92eb#C#compaction#133 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:01,782 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/5e08e8c15cf840e78344c87377efeae1 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/5e08e8c15cf840e78344c87377efeae1 2023-04-22 19:54:01,787 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/C of d6400a2b7b67145225eb12601c5e92eb into 5e08e8c15cf840e78344c87377efeae1(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:01,787 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:54:01,787 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/C, priority=13, startTime=1682193241727; duration=0sec 2023-04-22 19:54:01,788 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:02,153 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/76b6f319f07848d2966af9bcaa4ba7f5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/76b6f319f07848d2966af9bcaa4ba7f5 2023-04-22 19:54:02,158 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6400a2b7b67145225eb12601c5e92eb/A of d6400a2b7b67145225eb12601c5e92eb into 76b6f319f07848d2966af9bcaa4ba7f5(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:02,158 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:54:02,158 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb., storeName=d6400a2b7b67145225eb12601c5e92eb/A, priority=13, startTime=1682193241727; duration=0sec 2023-04-22 19:54:02,158 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:05,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:54:05,799 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-22 19:54:05,799 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:54:05,799 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:53381 2023-04-22 19:54:05,799 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:05,799 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:05,800 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:54:05,800 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:05,800 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:54:05,800 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:05,843 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61e72c30 to 127.0.0.1:53381 2023-04-22 19:54:05,843 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:05,843 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02df095a to 127.0.0.1:53381 2023-04-22 19:54:05,843 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:05,892 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:53381 2023-04-22 19:54:05,892 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:05,892 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 19:54:05,892 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 55 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 53 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 350 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 4017 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 3920 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1756 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 5268 rows 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1788 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 5364 rows 2023-04-22 19:54:05,893 INFO [Listener at localhost/46665] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 19:54:05,893 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6d61f291 to 127.0.0.1:53381 2023-04-22 19:54:05,894 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:05,900 INFO [Listener at localhost/46665] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 19:54:05,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 19:54:05,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:05,918 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193245918"}]},"ts":"1682193245918"} 2023-04-22 19:54:05,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 19:54:05,920 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 19:54:05,922 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 19:54:05,924 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6400a2b7b67145225eb12601c5e92eb, UNASSIGN}] 2023-04-22 19:54:05,926 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6400a2b7b67145225eb12601c5e92eb, UNASSIGN 2023-04-22 19:54:05,927 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=d6400a2b7b67145225eb12601c5e92eb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:05,928 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193245927"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193245927"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193245927"}]},"ts":"1682193245927"} 2023-04-22 19:54:05,930 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:54:06,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 19:54:06,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:54:06,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d6400a2b7b67145225eb12601c5e92eb, disabling compactions & flushes 2023-04-22 19:54:06,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:54:06,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/5a2f703a75d24bc2b3901791b5ebdd28 2023-04-22 19:54:06,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 19:54:06,223 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/5677f229bb7d425c9f972752de591858 2023-04-22 19:54:06,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 19:54:06,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/14caa8b7b9a643b7b2528d194e10cdd9 2023-04-22 19:54:06,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/5a2f703a75d24bc2b3901791b5ebdd28 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5a2f703a75d24bc2b3901791b5ebdd28 2023-04-22 19:54:06,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5a2f703a75d24bc2b3901791b5ebdd28, entries=150, sequenceid=681, filesize=11.9 K 2023-04-22 19:54:06,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/5677f229bb7d425c9f972752de591858 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5677f229bb7d425c9f972752de591858 2023-04-22 19:54:06,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5677f229bb7d425c9f972752de591858, entries=150, sequenceid=681, filesize=11.9 K 2023-04-22 19:54:06,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/14caa8b7b9a643b7b2528d194e10cdd9 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/14caa8b7b9a643b7b2528d194e10cdd9 2023-04-22 19:54:06,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/14caa8b7b9a643b7b2528d194e10cdd9, entries=150, sequenceid=681, filesize=11.9 K 2023-04-22 19:54:06,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=20.13 KB/20610 for d6400a2b7b67145225eb12601c5e92eb in 856ms, sequenceid=681, compaction requested=false 2023-04-22 19:54:06,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:54:06,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:54:06,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:54:06,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. after waiting 0 ms 2023-04-22 19:54:06,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:54:06,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d6400a2b7b67145225eb12601c5e92eb 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-22 19:54:06,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=A 2023-04-22 19:54:06,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:06,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=B 2023-04-22 19:54:06,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:06,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6400a2b7b67145225eb12601c5e92eb, store=C 2023-04-22 19:54:06,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:07,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 19:54:07,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/65301ca150994d8d90545aa60d98f555 2023-04-22 19:54:07,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/d081b674869046e198bf8345cad1aea2 2023-04-22 19:54:07,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/265d581c668341f18d27321589bc98fd 2023-04-22 19:54:07,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/A/65301ca150994d8d90545aa60d98f555 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/65301ca150994d8d90545aa60d98f555 2023-04-22 19:54:07,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/65301ca150994d8d90545aa60d98f555, entries=50, sequenceid=687, filesize=7.1 K 2023-04-22 19:54:07,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/B/d081b674869046e198bf8345cad1aea2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d081b674869046e198bf8345cad1aea2 2023-04-22 19:54:07,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d081b674869046e198bf8345cad1aea2, entries=50, sequenceid=687, filesize=7.1 K 2023-04-22 19:54:07,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/.tmp/C/265d581c668341f18d27321589bc98fd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/265d581c668341f18d27321589bc98fd 2023-04-22 19:54:07,911 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/265d581c668341f18d27321589bc98fd, entries=50, sequenceid=687, filesize=7.1 K 2023-04-22 19:54:07,912 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for d6400a2b7b67145225eb12601c5e92eb in 1257ms, sequenceid=687, compaction requested=true 2023-04-22 19:54:07,942 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9c85c7a9721f49b198557d5bde210c14, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/ca8c788a90aa4972a1db1e455600fe6b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f3182230e77e408bbcdaacfeae2033ee, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/d9d3787e205146ebb3ae6ab11b8cbebc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/082da84dc6ad4a0e9a8114ed538670f2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/27ca87b19f454df48ce0ff84db084e0a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/8d3f17b8681e415fbcd4279ba6b2f953, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/cf603db045234b63af0be26f533d5b81, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/fa8da5f8ea8c4aa4843b3291488f83f7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1e3b13be5b6843cf9d0327ec82d916e6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f1fb8e9b5663431ca6b2580f451cb17c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/427d1bcae0fc488481aab28986f8d9a3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/56ee2bc4e4ad48318fec22ff0d4e607a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/2ddb5cbf91f046edaaddd1158c62b3a5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1caca770e1e445eabd364f48d73f10ae, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/e48b1f11a3594038a001cc6e44b54aa7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/bf84ac2ac7c748d98c1d940897a1cc7b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1fbf739a289f4b3ca20082aad2721167, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6521f931af4e4f3dae67b718c852e339, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5816344e8b4d4588b8ef246fe865854f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/65a1e10706df43e5ae9e84fc3e487b3a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/15e1a54631064394bcfb9f97db9516ae, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6bdc6e679f104db8a56fa9ffe0e5266a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/78a35cf9fc2d4e699291b09cfbaab63e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/79ce90e6974347299db7d076317f0ada, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5e0ca88f7b3e4cc69aafaa52d2f0fbe1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/c276f28f3efb4ffdad9419c0612cbbc3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/81f18a55af28462a897d9ad91e3a8fa9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/b650032a513d45c0ae2e61e20d18fbdc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/dcfc1428b0304f91b2b330f5afb04fa2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/3bbf81cd303049fdbc7d873a2ffb1e95, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/198bdae3345143639d6e9c62d6753e2d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a8615b96b7434801bd46c44a2b68241f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9f4a0acda2cd48178c2e56541330f4ca, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5c5fc24b23d949d69a5b8352b05d9696, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/60c2e58d677247d9b3d4ddf99b8d640e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e34c891a86c49c8b942f9cccca25ad6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e666c8737854fc2a16d868f7bf9ba28, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/79783a9a97204b0d850c2b3e2a10e612, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/01a6239f38e24158bc6f6920ea4fb312, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a6100e48691647759db7133d071068bf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1382fd68715445ef94ef11babc69c4a3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/273ad6fd24d94584a81572bd62782f0e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/909c96ab85904150a3177df753552bec] to archive 2023-04-22 19:54:07,943 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:54:07,953 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9c85c7a9721f49b198557d5bde210c14 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9c85c7a9721f49b198557d5bde210c14 2023-04-22 19:54:07,955 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/ca8c788a90aa4972a1db1e455600fe6b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/ca8c788a90aa4972a1db1e455600fe6b 2023-04-22 19:54:07,956 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f3182230e77e408bbcdaacfeae2033ee to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f3182230e77e408bbcdaacfeae2033ee 2023-04-22 19:54:07,958 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/d9d3787e205146ebb3ae6ab11b8cbebc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/d9d3787e205146ebb3ae6ab11b8cbebc 2023-04-22 19:54:07,961 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/082da84dc6ad4a0e9a8114ed538670f2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/082da84dc6ad4a0e9a8114ed538670f2 2023-04-22 19:54:07,965 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/27ca87b19f454df48ce0ff84db084e0a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/27ca87b19f454df48ce0ff84db084e0a 2023-04-22 19:54:07,966 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/8d3f17b8681e415fbcd4279ba6b2f953 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/8d3f17b8681e415fbcd4279ba6b2f953 2023-04-22 19:54:07,967 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/cf603db045234b63af0be26f533d5b81 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/cf603db045234b63af0be26f533d5b81 2023-04-22 19:54:07,969 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/fa8da5f8ea8c4aa4843b3291488f83f7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/fa8da5f8ea8c4aa4843b3291488f83f7 2023-04-22 19:54:07,970 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1e3b13be5b6843cf9d0327ec82d916e6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1e3b13be5b6843cf9d0327ec82d916e6 2023-04-22 19:54:07,971 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f1fb8e9b5663431ca6b2580f451cb17c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/f1fb8e9b5663431ca6b2580f451cb17c 2023-04-22 19:54:07,972 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/427d1bcae0fc488481aab28986f8d9a3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/427d1bcae0fc488481aab28986f8d9a3 2023-04-22 19:54:07,974 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/56ee2bc4e4ad48318fec22ff0d4e607a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/56ee2bc4e4ad48318fec22ff0d4e607a 2023-04-22 19:54:07,975 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/2ddb5cbf91f046edaaddd1158c62b3a5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/2ddb5cbf91f046edaaddd1158c62b3a5 2023-04-22 19:54:07,976 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1caca770e1e445eabd364f48d73f10ae to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1caca770e1e445eabd364f48d73f10ae 2023-04-22 19:54:07,978 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/e48b1f11a3594038a001cc6e44b54aa7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/e48b1f11a3594038a001cc6e44b54aa7 2023-04-22 19:54:07,979 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/bf84ac2ac7c748d98c1d940897a1cc7b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/bf84ac2ac7c748d98c1d940897a1cc7b 2023-04-22 19:54:07,980 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1fbf739a289f4b3ca20082aad2721167 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1fbf739a289f4b3ca20082aad2721167 2023-04-22 19:54:07,981 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6521f931af4e4f3dae67b718c852e339 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6521f931af4e4f3dae67b718c852e339 2023-04-22 19:54:07,982 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5816344e8b4d4588b8ef246fe865854f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5816344e8b4d4588b8ef246fe865854f 2023-04-22 19:54:07,983 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/65a1e10706df43e5ae9e84fc3e487b3a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/65a1e10706df43e5ae9e84fc3e487b3a 2023-04-22 19:54:07,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/15e1a54631064394bcfb9f97db9516ae to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/15e1a54631064394bcfb9f97db9516ae 2023-04-22 19:54:07,986 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6bdc6e679f104db8a56fa9ffe0e5266a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/6bdc6e679f104db8a56fa9ffe0e5266a 2023-04-22 19:54:07,987 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/78a35cf9fc2d4e699291b09cfbaab63e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/78a35cf9fc2d4e699291b09cfbaab63e 2023-04-22 19:54:07,989 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/79ce90e6974347299db7d076317f0ada to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/79ce90e6974347299db7d076317f0ada 2023-04-22 19:54:07,990 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5e0ca88f7b3e4cc69aafaa52d2f0fbe1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5e0ca88f7b3e4cc69aafaa52d2f0fbe1 2023-04-22 19:54:07,991 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/c276f28f3efb4ffdad9419c0612cbbc3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/c276f28f3efb4ffdad9419c0612cbbc3 2023-04-22 19:54:07,992 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/81f18a55af28462a897d9ad91e3a8fa9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/81f18a55af28462a897d9ad91e3a8fa9 2023-04-22 19:54:07,993 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/b650032a513d45c0ae2e61e20d18fbdc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/b650032a513d45c0ae2e61e20d18fbdc 2023-04-22 19:54:07,994 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/dcfc1428b0304f91b2b330f5afb04fa2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/dcfc1428b0304f91b2b330f5afb04fa2 2023-04-22 19:54:07,996 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/3bbf81cd303049fdbc7d873a2ffb1e95 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/3bbf81cd303049fdbc7d873a2ffb1e95 2023-04-22 19:54:07,997 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/198bdae3345143639d6e9c62d6753e2d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/198bdae3345143639d6e9c62d6753e2d 2023-04-22 19:54:07,998 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a8615b96b7434801bd46c44a2b68241f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a8615b96b7434801bd46c44a2b68241f 2023-04-22 19:54:07,999 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9f4a0acda2cd48178c2e56541330f4ca to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/9f4a0acda2cd48178c2e56541330f4ca 2023-04-22 19:54:08,000 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5c5fc24b23d949d69a5b8352b05d9696 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5c5fc24b23d949d69a5b8352b05d9696 2023-04-22 19:54:08,001 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/60c2e58d677247d9b3d4ddf99b8d640e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/60c2e58d677247d9b3d4ddf99b8d640e 2023-04-22 19:54:08,002 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e34c891a86c49c8b942f9cccca25ad6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e34c891a86c49c8b942f9cccca25ad6 2023-04-22 19:54:08,003 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e666c8737854fc2a16d868f7bf9ba28 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/7e666c8737854fc2a16d868f7bf9ba28 2023-04-22 19:54:08,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/79783a9a97204b0d850c2b3e2a10e612 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/79783a9a97204b0d850c2b3e2a10e612 2023-04-22 19:54:08,005 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/01a6239f38e24158bc6f6920ea4fb312 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/01a6239f38e24158bc6f6920ea4fb312 2023-04-22 19:54:08,006 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a6100e48691647759db7133d071068bf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/a6100e48691647759db7133d071068bf 2023-04-22 19:54:08,007 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1382fd68715445ef94ef11babc69c4a3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/1382fd68715445ef94ef11babc69c4a3 2023-04-22 19:54:08,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/273ad6fd24d94584a81572bd62782f0e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/273ad6fd24d94584a81572bd62782f0e 2023-04-22 19:54:08,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/909c96ab85904150a3177df753552bec to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/909c96ab85904150a3177df753552bec 2023-04-22 19:54:08,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 19:54:08,035 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/92e7c5db8d88425782c436c1e6dd3aac, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/1bece0325c964ab7acd939ecdac15313, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0d4d87daac5a4d7e8275c12f1b239afc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/22f1e7236a71456f8c497a8440d1636f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/bad04121107948c591582c74826e03a2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/641a8e290b0b45c1abbbb1dcf1e88855, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5264011cebfa46a4a1df7073e37722e4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/ead8ccd4e9414f9c9ad01aed4b5bf706, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/de6e4e2affb94378b90d37ccdc7f6f15, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/25bb682bdb8a4d3dba3c330345d48774, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/684321cbf38c445b948a61f2a9810520, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/f2ddf535203b4832bef69842d7a187a0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/09881ea805334fcea821bc95e098d300, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/65fc155c669f4639a8445a583b84f3f9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/dc05e87821e84dce93f82a8dd65446d8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e2404d8178234ec98b4d8db3b116318d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d0deefc0e7bf41d0b44ac3d4282cf90a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b8e16d0401fc4642992e282798780688, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/9c1fbe2ef79242f3bc52ee2dfe600ec0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/6405866964b846f2ae35d2ed92587632, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e89d9b1d198a4364b1cca0845f59948f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5d283e1e607d4d0bad8c286267d31219, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/21c5b2d7ea6d4c23923fbbc4de7c7efd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d012860eded04c17a9959a07d73bb0af, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b4abd2104a224688b157df112b83d600, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/3f77722a84fe422bb54030587af86566, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c2febc5efbbb4f11a0f1a77d5c6a71e0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0da7fb150f5b4eb1855871fdb5566c24, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/8d8944a9b56747eb841caa46d3326526, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a973d36663484727a910341d990e11d9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0dcb3546aca042deacaa9bd5522c49dd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b44c60f8f61949f9ac664e070123d23e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0c4efd388ae94c6e9efb42c5b7634580, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e0953d2637614f728cda618239cf2aa5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c84d9f72547d49908543790a85606c9a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a1957feecd024b149a3102820ba0b32c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7c923c40217e4298b4a614187d730767, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/45783de6a9564298ba7ef564c3d3a1f1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7e1065fd155d4abca116bc4ca501e0d7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/691be9325f2b498ca90ddd20f4c626da, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0668bea639104f5389ef7fba0e83bd40, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c3c64310097c4a43b787382451aba65a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/26f32f18372648e9be0976ed86d52eaa, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/47fd7720795045859d4af0d0620c6ced] to archive 2023-04-22 19:54:08,036 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:54:08,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/92e7c5db8d88425782c436c1e6dd3aac to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/92e7c5db8d88425782c436c1e6dd3aac 2023-04-22 19:54:08,039 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/1bece0325c964ab7acd939ecdac15313 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/1bece0325c964ab7acd939ecdac15313 2023-04-22 19:54:08,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0d4d87daac5a4d7e8275c12f1b239afc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0d4d87daac5a4d7e8275c12f1b239afc 2023-04-22 19:54:08,042 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/22f1e7236a71456f8c497a8440d1636f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/22f1e7236a71456f8c497a8440d1636f 2023-04-22 19:54:08,043 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/bad04121107948c591582c74826e03a2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/bad04121107948c591582c74826e03a2 2023-04-22 19:54:08,044 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/641a8e290b0b45c1abbbb1dcf1e88855 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/641a8e290b0b45c1abbbb1dcf1e88855 2023-04-22 19:54:08,045 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5264011cebfa46a4a1df7073e37722e4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5264011cebfa46a4a1df7073e37722e4 2023-04-22 19:54:08,047 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/ead8ccd4e9414f9c9ad01aed4b5bf706 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/ead8ccd4e9414f9c9ad01aed4b5bf706 2023-04-22 19:54:08,048 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/de6e4e2affb94378b90d37ccdc7f6f15 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/de6e4e2affb94378b90d37ccdc7f6f15 2023-04-22 19:54:08,049 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/25bb682bdb8a4d3dba3c330345d48774 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/25bb682bdb8a4d3dba3c330345d48774 2023-04-22 19:54:08,050 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/684321cbf38c445b948a61f2a9810520 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/684321cbf38c445b948a61f2a9810520 2023-04-22 19:54:08,051 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/f2ddf535203b4832bef69842d7a187a0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/f2ddf535203b4832bef69842d7a187a0 2023-04-22 19:54:08,053 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/09881ea805334fcea821bc95e098d300 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/09881ea805334fcea821bc95e098d300 2023-04-22 19:54:08,054 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/65fc155c669f4639a8445a583b84f3f9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/65fc155c669f4639a8445a583b84f3f9 2023-04-22 19:54:08,056 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/dc05e87821e84dce93f82a8dd65446d8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/dc05e87821e84dce93f82a8dd65446d8 2023-04-22 19:54:08,057 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e2404d8178234ec98b4d8db3b116318d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e2404d8178234ec98b4d8db3b116318d 2023-04-22 19:54:08,059 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d0deefc0e7bf41d0b44ac3d4282cf90a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d0deefc0e7bf41d0b44ac3d4282cf90a 2023-04-22 19:54:08,060 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b8e16d0401fc4642992e282798780688 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b8e16d0401fc4642992e282798780688 2023-04-22 19:54:08,062 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/9c1fbe2ef79242f3bc52ee2dfe600ec0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/9c1fbe2ef79242f3bc52ee2dfe600ec0 2023-04-22 19:54:08,063 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/6405866964b846f2ae35d2ed92587632 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/6405866964b846f2ae35d2ed92587632 2023-04-22 19:54:08,065 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e89d9b1d198a4364b1cca0845f59948f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e89d9b1d198a4364b1cca0845f59948f 2023-04-22 19:54:08,067 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5d283e1e607d4d0bad8c286267d31219 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5d283e1e607d4d0bad8c286267d31219 2023-04-22 19:54:08,069 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/21c5b2d7ea6d4c23923fbbc4de7c7efd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/21c5b2d7ea6d4c23923fbbc4de7c7efd 2023-04-22 19:54:08,071 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d012860eded04c17a9959a07d73bb0af to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d012860eded04c17a9959a07d73bb0af 2023-04-22 19:54:08,073 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b4abd2104a224688b157df112b83d600 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b4abd2104a224688b157df112b83d600 2023-04-22 19:54:08,075 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/3f77722a84fe422bb54030587af86566 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/3f77722a84fe422bb54030587af86566 2023-04-22 19:54:08,076 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c2febc5efbbb4f11a0f1a77d5c6a71e0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c2febc5efbbb4f11a0f1a77d5c6a71e0 2023-04-22 19:54:08,078 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0da7fb150f5b4eb1855871fdb5566c24 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0da7fb150f5b4eb1855871fdb5566c24 2023-04-22 19:54:08,079 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/8d8944a9b56747eb841caa46d3326526 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/8d8944a9b56747eb841caa46d3326526 2023-04-22 19:54:08,081 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a973d36663484727a910341d990e11d9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a973d36663484727a910341d990e11d9 2023-04-22 19:54:08,082 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0dcb3546aca042deacaa9bd5522c49dd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0dcb3546aca042deacaa9bd5522c49dd 2023-04-22 19:54:08,083 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b44c60f8f61949f9ac664e070123d23e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/b44c60f8f61949f9ac664e070123d23e 2023-04-22 19:54:08,085 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0c4efd388ae94c6e9efb42c5b7634580 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0c4efd388ae94c6e9efb42c5b7634580 2023-04-22 19:54:08,086 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e0953d2637614f728cda618239cf2aa5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/e0953d2637614f728cda618239cf2aa5 2023-04-22 19:54:08,087 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c84d9f72547d49908543790a85606c9a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c84d9f72547d49908543790a85606c9a 2023-04-22 19:54:08,089 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a1957feecd024b149a3102820ba0b32c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/a1957feecd024b149a3102820ba0b32c 2023-04-22 19:54:08,090 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7c923c40217e4298b4a614187d730767 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7c923c40217e4298b4a614187d730767 2023-04-22 19:54:08,091 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/45783de6a9564298ba7ef564c3d3a1f1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/45783de6a9564298ba7ef564c3d3a1f1 2023-04-22 19:54:08,092 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7e1065fd155d4abca116bc4ca501e0d7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/7e1065fd155d4abca116bc4ca501e0d7 2023-04-22 19:54:08,094 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/691be9325f2b498ca90ddd20f4c626da to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/691be9325f2b498ca90ddd20f4c626da 2023-04-22 19:54:08,095 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0668bea639104f5389ef7fba0e83bd40 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/0668bea639104f5389ef7fba0e83bd40 2023-04-22 19:54:08,096 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c3c64310097c4a43b787382451aba65a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/c3c64310097c4a43b787382451aba65a 2023-04-22 19:54:08,098 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/26f32f18372648e9be0976ed86d52eaa to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/26f32f18372648e9be0976ed86d52eaa 2023-04-22 19:54:08,099 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/47fd7720795045859d4af0d0620c6ced to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/47fd7720795045859d4af0d0620c6ced 2023-04-22 19:54:08,103 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3801ce1ef3334bcaafef83545c8cce63, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c4db1382afac4193af6564ba8476bd0d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/d51cc055c9b54feb87a95fde7b5f7d00, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7b1bdf33392345bc876396da603e09ca, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/97238151514d4ffaad5052995001f9c4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3a470c83c618495a91a5aa106b6c18bf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c254d8d884b944918b8e9681712ee290, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/fef0509ad33a4fa9951209ae2c8c001c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/48f37ef3063740b295fe4d7cf4f60463, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/5f99f7ce28234219964fc91e853b75c7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/070674eaf90a412caeb261f6dde3fff3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b07566910d704fe3bc46cdc9e574212c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7cdae021a3034378a84f609c07e5902e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/89711432d3a0497fb9c4f3fce9bfddc0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/4876531ed8dc4da7a6a84d91bb0dd60b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/233bbf70d5b54bb899ddbbacda9b1329, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b541c3230bdb410aac4f4d31ef2156d7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/bda9dcebfb1446c2ba23f198e09e931b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/87767bd4d27d40339fca18a463373450, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1e7a87fe871742b2b5972dd38aff1091, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c2e97959ad164b57baa5b475e5bf241c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0fd5c38ebb91470e8b5f93d817a32e81, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0a5263ac8ed74d938a199dfab7d38e5a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2ef56d4f783346a29a9a4d8635e3c9a7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1f978d9f8abe4a358049b9d082bdadc5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/58e7f21ea0f24898a219348f1eceac6c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/726ce34271674d6399213fb53cd7168d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/a1c0166a7118408b96074f8c90b826d7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f6be02e773f94a9aae5e2606e173f3b8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1aca6123e6c14a629282337d251707f3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/8478acaace4848b5b571949462fbf222, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/fbe334ce4cee4c5484b6540847296219, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7a205c37f8534dc7a05e34ac13b6dc53, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/489919b6cc9a417ca9d708e818607dd4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c605e0bb174749968fd3735beffb4276, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/de76b7ef4db740ab92faa35affebd069, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1dc7525130e84f51ae205ce5fbaf67c7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3b681b08bf7a4e0daa067b4be01003f8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/56b271f6a1e4453a8da4ed4513a18b1d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/26f7d76cf93f485597d9838375dc6a63, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f75cc290bb7f4eb4a8997716362f5f63, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/10d890511baf4680b035a197a2cb7cae, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2154edfcb4b0405bbb755b8366af602a] to archive 2023-04-22 19:54:08,104 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:54:08,106 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3801ce1ef3334bcaafef83545c8cce63 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3801ce1ef3334bcaafef83545c8cce63 2023-04-22 19:54:08,107 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c4db1382afac4193af6564ba8476bd0d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c4db1382afac4193af6564ba8476bd0d 2023-04-22 19:54:08,109 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/d51cc055c9b54feb87a95fde7b5f7d00 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/d51cc055c9b54feb87a95fde7b5f7d00 2023-04-22 19:54:08,110 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7b1bdf33392345bc876396da603e09ca to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7b1bdf33392345bc876396da603e09ca 2023-04-22 19:54:08,111 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/97238151514d4ffaad5052995001f9c4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/97238151514d4ffaad5052995001f9c4 2023-04-22 19:54:08,113 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3a470c83c618495a91a5aa106b6c18bf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3a470c83c618495a91a5aa106b6c18bf 2023-04-22 19:54:08,114 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c254d8d884b944918b8e9681712ee290 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c254d8d884b944918b8e9681712ee290 2023-04-22 19:54:08,115 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/fef0509ad33a4fa9951209ae2c8c001c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/fef0509ad33a4fa9951209ae2c8c001c 2023-04-22 19:54:08,116 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/48f37ef3063740b295fe4d7cf4f60463 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/48f37ef3063740b295fe4d7cf4f60463 2023-04-22 19:54:08,117 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/5f99f7ce28234219964fc91e853b75c7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/5f99f7ce28234219964fc91e853b75c7 2023-04-22 19:54:08,118 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/070674eaf90a412caeb261f6dde3fff3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/070674eaf90a412caeb261f6dde3fff3 2023-04-22 19:54:08,120 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b07566910d704fe3bc46cdc9e574212c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b07566910d704fe3bc46cdc9e574212c 2023-04-22 19:54:08,121 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7cdae021a3034378a84f609c07e5902e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7cdae021a3034378a84f609c07e5902e 2023-04-22 19:54:08,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/89711432d3a0497fb9c4f3fce9bfddc0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/89711432d3a0497fb9c4f3fce9bfddc0 2023-04-22 19:54:08,125 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/4876531ed8dc4da7a6a84d91bb0dd60b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/4876531ed8dc4da7a6a84d91bb0dd60b 2023-04-22 19:54:08,126 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/233bbf70d5b54bb899ddbbacda9b1329 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/233bbf70d5b54bb899ddbbacda9b1329 2023-04-22 19:54:08,127 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b541c3230bdb410aac4f4d31ef2156d7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/b541c3230bdb410aac4f4d31ef2156d7 2023-04-22 19:54:08,128 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/bda9dcebfb1446c2ba23f198e09e931b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/bda9dcebfb1446c2ba23f198e09e931b 2023-04-22 19:54:08,130 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/87767bd4d27d40339fca18a463373450 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/87767bd4d27d40339fca18a463373450 2023-04-22 19:54:08,131 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1e7a87fe871742b2b5972dd38aff1091 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1e7a87fe871742b2b5972dd38aff1091 2023-04-22 19:54:08,133 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c2e97959ad164b57baa5b475e5bf241c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c2e97959ad164b57baa5b475e5bf241c 2023-04-22 19:54:08,134 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0fd5c38ebb91470e8b5f93d817a32e81 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0fd5c38ebb91470e8b5f93d817a32e81 2023-04-22 19:54:08,136 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0a5263ac8ed74d938a199dfab7d38e5a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/0a5263ac8ed74d938a199dfab7d38e5a 2023-04-22 19:54:08,137 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2ef56d4f783346a29a9a4d8635e3c9a7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2ef56d4f783346a29a9a4d8635e3c9a7 2023-04-22 19:54:08,138 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1f978d9f8abe4a358049b9d082bdadc5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1f978d9f8abe4a358049b9d082bdadc5 2023-04-22 19:54:08,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/58e7f21ea0f24898a219348f1eceac6c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/58e7f21ea0f24898a219348f1eceac6c 2023-04-22 19:54:08,140 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/726ce34271674d6399213fb53cd7168d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/726ce34271674d6399213fb53cd7168d 2023-04-22 19:54:08,141 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/a1c0166a7118408b96074f8c90b826d7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/a1c0166a7118408b96074f8c90b826d7 2023-04-22 19:54:08,142 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f6be02e773f94a9aae5e2606e173f3b8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f6be02e773f94a9aae5e2606e173f3b8 2023-04-22 19:54:08,143 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1aca6123e6c14a629282337d251707f3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1aca6123e6c14a629282337d251707f3 2023-04-22 19:54:08,145 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/8478acaace4848b5b571949462fbf222 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/8478acaace4848b5b571949462fbf222 2023-04-22 19:54:08,146 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/fbe334ce4cee4c5484b6540847296219 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/fbe334ce4cee4c5484b6540847296219 2023-04-22 19:54:08,147 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7a205c37f8534dc7a05e34ac13b6dc53 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/7a205c37f8534dc7a05e34ac13b6dc53 2023-04-22 19:54:08,148 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/489919b6cc9a417ca9d708e818607dd4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/489919b6cc9a417ca9d708e818607dd4 2023-04-22 19:54:08,149 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c605e0bb174749968fd3735beffb4276 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/c605e0bb174749968fd3735beffb4276 2023-04-22 19:54:08,151 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/de76b7ef4db740ab92faa35affebd069 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/de76b7ef4db740ab92faa35affebd069 2023-04-22 19:54:08,152 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1dc7525130e84f51ae205ce5fbaf67c7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/1dc7525130e84f51ae205ce5fbaf67c7 2023-04-22 19:54:08,153 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3b681b08bf7a4e0daa067b4be01003f8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/3b681b08bf7a4e0daa067b4be01003f8 2023-04-22 19:54:08,154 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/56b271f6a1e4453a8da4ed4513a18b1d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/56b271f6a1e4453a8da4ed4513a18b1d 2023-04-22 19:54:08,155 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/26f7d76cf93f485597d9838375dc6a63 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/26f7d76cf93f485597d9838375dc6a63 2023-04-22 19:54:08,157 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f75cc290bb7f4eb4a8997716362f5f63 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/f75cc290bb7f4eb4a8997716362f5f63 2023-04-22 19:54:08,158 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/10d890511baf4680b035a197a2cb7cae to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/10d890511baf4680b035a197a2cb7cae 2023-04-22 19:54:08,160 DEBUG [StoreCloser-TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2154edfcb4b0405bbb755b8366af602a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/2154edfcb4b0405bbb755b8366af602a 2023-04-22 19:54:08,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/recovered.edits/690.seqid, newMaxSeqId=690, maxSeqId=1 2023-04-22 19:54:08,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb. 2023-04-22 19:54:08,170 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d6400a2b7b67145225eb12601c5e92eb: 2023-04-22 19:54:08,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:54:08,172 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=d6400a2b7b67145225eb12601c5e92eb, regionState=CLOSED 2023-04-22 19:54:08,172 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193248172"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193248172"}]},"ts":"1682193248172"} 2023-04-22 19:54:08,176 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-22 19:54:08,176 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure d6400a2b7b67145225eb12601c5e92eb, server=jenkins-hbase4.apache.org,44791,1682193215766 in 2.2440 sec 2023-04-22 19:54:08,179 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-22 19:54:08,179 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6400a2b7b67145225eb12601c5e92eb, UNASSIGN in 2.2520 sec 2023-04-22 19:54:08,180 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193248180"}]},"ts":"1682193248180"} 2023-04-22 19:54:08,181 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 19:54:08,183 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 19:54:08,185 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.2750 sec 2023-04-22 19:54:10,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 19:54:10,026 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-22 19:54:10,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 19:54:10,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:10,038 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:10,040 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:10,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-22 19:54:10,045 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:54:10,048 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/recovered.edits] 2023-04-22 19:54:10,051 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5a2f703a75d24bc2b3901791b5ebdd28 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/5a2f703a75d24bc2b3901791b5ebdd28 2023-04-22 19:54:10,052 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/65301ca150994d8d90545aa60d98f555 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/65301ca150994d8d90545aa60d98f555 2023-04-22 19:54:10,053 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/76b6f319f07848d2966af9bcaa4ba7f5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/A/76b6f319f07848d2966af9bcaa4ba7f5 2023-04-22 19:54:10,055 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5677f229bb7d425c9f972752de591858 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/5677f229bb7d425c9f972752de591858 2023-04-22 19:54:10,057 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/9e18195bc38748aca4e0b21c5b013145 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/9e18195bc38748aca4e0b21c5b013145 2023-04-22 19:54:10,058 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d081b674869046e198bf8345cad1aea2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/B/d081b674869046e198bf8345cad1aea2 2023-04-22 19:54:10,060 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/14caa8b7b9a643b7b2528d194e10cdd9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/14caa8b7b9a643b7b2528d194e10cdd9 2023-04-22 19:54:10,061 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/265d581c668341f18d27321589bc98fd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/265d581c668341f18d27321589bc98fd 2023-04-22 19:54:10,062 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/5e08e8c15cf840e78344c87377efeae1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/C/5e08e8c15cf840e78344c87377efeae1 2023-04-22 19:54:10,065 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/recovered.edits/690.seqid to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb/recovered.edits/690.seqid 2023-04-22 19:54:10,066 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/d6400a2b7b67145225eb12601c5e92eb 2023-04-22 19:54:10,066 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:54:10,069 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:10,088 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 19:54:10,132 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 19:54:10,133 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:10,134 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 19:54:10,134 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682193250134"}]},"ts":"9223372036854775807"} 2023-04-22 19:54:10,137 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 19:54:10,137 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d6400a2b7b67145225eb12601c5e92eb, NAME => 'TestAcidGuarantees,,1682193219747.d6400a2b7b67145225eb12601c5e92eb.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 19:54:10,137 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 19:54:10,137 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682193250137"}]},"ts":"9223372036854775807"} 2023-04-22 19:54:10,139 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 19:54:10,141 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:10,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-22 19:54:10,143 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 109 msec 2023-04-22 19:54:10,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-22 19:54:10,344 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-22 19:54:10,360 INFO [Listener at localhost/46665] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=231 (was 206) Potentially hanging thread: RS:0;jenkins-hbase4:44791-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7d62d032-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7d62d032-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_8413944_15 at /127.0.0.1:42928 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_8413944_15 at /127.0.0.1:43002 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7d62d032-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7d62d032-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=576 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=413 (was 359) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=12669 (was 13268) 2023-04-22 19:54:10,372 INFO [Listener at localhost/46665] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=231, OpenFileDescriptor=576, MaxFileDescriptor=60000, SystemLoadAverage=413, ProcessCount=169, AvailableMemoryMB=12668 2023-04-22 19:54:10,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 19:54:10,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 19:54:10,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:10,376 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 19:54:10,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-22 19:54:10,377 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 19:54:10,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 19:54:10,379 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:10,379 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11 empty. 2023-04-22 19:54:10,380 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:10,380 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:54:10,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 19:54:10,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 19:54:10,788 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 19:54:10,790 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 98809de8ceb9ac8f3bc773e820fd0e11, NAME => 'TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp 2023-04-22 19:54:10,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 19:54:11,196 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:54:11,196 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 98809de8ceb9ac8f3bc773e820fd0e11, disabling compactions & flushes 2023-04-22 19:54:11,196 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:11,196 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:11,196 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. after waiting 0 ms 2023-04-22 19:54:11,196 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:11,196 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:11,196 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:11,200 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 19:54:11,201 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193251201"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193251201"}]},"ts":"1682193251201"} 2023-04-22 19:54:11,205 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 19:54:11,205 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 19:54:11,206 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193251206"}]},"ts":"1682193251206"} 2023-04-22 19:54:11,208 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 19:54:11,219 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=98809de8ceb9ac8f3bc773e820fd0e11, ASSIGN}] 2023-04-22 19:54:11,221 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=98809de8ceb9ac8f3bc773e820fd0e11, ASSIGN 2023-04-22 19:54:11,222 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=98809de8ceb9ac8f3bc773e820fd0e11, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44791,1682193215766; forceNewPlan=false, retain=false 2023-04-22 19:54:11,373 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=98809de8ceb9ac8f3bc773e820fd0e11, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:11,373 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193251373"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193251373"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193251373"}]},"ts":"1682193251373"} 2023-04-22 19:54:11,375 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:54:11,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 19:54:11,531 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:11,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 98809de8ceb9ac8f3bc773e820fd0e11, NAME => 'TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:54:11,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:11,532 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:54:11,532 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:11,532 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:11,533 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:11,535 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A 2023-04-22 19:54:11,535 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A 2023-04-22 19:54:11,535 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:54:11,535 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98809de8ceb9ac8f3bc773e820fd0e11 columnFamilyName A 2023-04-22 19:54:11,536 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(310): Store=98809de8ceb9ac8f3bc773e820fd0e11/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:54:11,536 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:11,537 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B 2023-04-22 19:54:11,537 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B 2023-04-22 19:54:11,537 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:54:11,537 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98809de8ceb9ac8f3bc773e820fd0e11 columnFamilyName B 2023-04-22 19:54:11,538 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(310): Store=98809de8ceb9ac8f3bc773e820fd0e11/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:54:11,538 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:11,539 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C 2023-04-22 19:54:11,539 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C 2023-04-22 19:54:11,539 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:54:11,540 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98809de8ceb9ac8f3bc773e820fd0e11 columnFamilyName C 2023-04-22 19:54:11,540 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(310): Store=98809de8ceb9ac8f3bc773e820fd0e11/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:54:11,540 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:11,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:11,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:11,543 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 19:54:11,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:11,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 19:54:11,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 98809de8ceb9ac8f3bc773e820fd0e11; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63442875, jitterRate=-0.054627493023872375}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 19:54:11,547 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:11,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., pid=18, masterSystemTime=1682193251527 2023-04-22 19:54:11,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:11,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:11,550 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=98809de8ceb9ac8f3bc773e820fd0e11, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:11,550 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193251549"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682193251549"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682193251549"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682193251549"}]},"ts":"1682193251549"} 2023-04-22 19:54:11,553 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-22 19:54:11,553 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 in 176 msec 2023-04-22 19:54:11,555 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-22 19:54:11,555 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=98809de8ceb9ac8f3bc773e820fd0e11, ASSIGN in 334 msec 2023-04-22 19:54:11,556 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 19:54:11,556 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193251556"}]},"ts":"1682193251556"} 2023-04-22 19:54:11,557 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 19:54:11,559 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 19:54:11,561 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1850 sec 2023-04-22 19:54:12,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 19:54:12,483 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-22 19:54:12,484 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x5c16875c to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:12,490 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a607873, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:12,492 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:12,496 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60366, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:12,497 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 19:54:12,500 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 19:54:12,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 19:54:12,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 19:54:12,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:12,932 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 19:54:12,933 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 19:54:12,933 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 19:54:12,937 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-22 19:54:12,949 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=98809de8ceb9ac8f3bc773e820fd0e11, REOPEN/MOVE}] 2023-04-22 19:54:12,950 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=98809de8ceb9ac8f3bc773e820fd0e11, REOPEN/MOVE 2023-04-22 19:54:12,951 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=98809de8ceb9ac8f3bc773e820fd0e11, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:12,951 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193252951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193252951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193252951"}]},"ts":"1682193252951"} 2023-04-22 19:54:12,952 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:54:13,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 98809de8ceb9ac8f3bc773e820fd0e11, disabling compactions & flushes 2023-04-22 19:54:13,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. after waiting 0 ms 2023-04-22 19:54:13,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-22 19:54:13,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:13,110 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 98809de8ceb9ac8f3bc773e820fd0e11 to self. 2023-04-22 19:54:13,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,112 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=98809de8ceb9ac8f3bc773e820fd0e11, regionState=CLOSED 2023-04-22 19:54:13,112 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193253112"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193253112"}]},"ts":"1682193253112"} 2023-04-22 19:54:13,114 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-22 19:54:13,115 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 in 161 msec 2023-04-22 19:54:13,115 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=98809de8ceb9ac8f3bc773e820fd0e11, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,44791,1682193215766; forceNewPlan=false, retain=true 2023-04-22 19:54:13,266 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=98809de8ceb9ac8f3bc773e820fd0e11, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,266 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193253265"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193253265"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193253265"}]},"ts":"1682193253265"} 2023-04-22 19:54:13,267 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:54:13,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 98809de8ceb9ac8f3bc773e820fd0e11, NAME => 'TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:54:13,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:54:13,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,425 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,426 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A 2023-04-22 19:54:13,426 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A 2023-04-22 19:54:13,427 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:54:13,429 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98809de8ceb9ac8f3bc773e820fd0e11 columnFamilyName A 2023-04-22 19:54:13,431 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(310): Store=98809de8ceb9ac8f3bc773e820fd0e11/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:54:13,431 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,432 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B 2023-04-22 19:54:13,432 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B 2023-04-22 19:54:13,433 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:54:13,433 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98809de8ceb9ac8f3bc773e820fd0e11 columnFamilyName B 2023-04-22 19:54:13,433 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(310): Store=98809de8ceb9ac8f3bc773e820fd0e11/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:54:13,434 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,434 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C 2023-04-22 19:54:13,434 DEBUG [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C 2023-04-22 19:54:13,434 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:54:13,435 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98809de8ceb9ac8f3bc773e820fd0e11 columnFamilyName C 2023-04-22 19:54:13,436 INFO [StoreOpener-98809de8ceb9ac8f3bc773e820fd0e11-1] regionserver.HStore(310): Store=98809de8ceb9ac8f3bc773e820fd0e11/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:54:13,436 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 19:54:13,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 98809de8ceb9ac8f3bc773e820fd0e11; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67329291, jitterRate=0.0032846182584762573}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 19:54:13,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:13,442 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., pid=23, masterSystemTime=1682193253419 2023-04-22 19:54:13,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,444 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=98809de8ceb9ac8f3bc773e820fd0e11, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,444 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193253444"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682193253444"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682193253444"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682193253444"}]},"ts":"1682193253444"} 2023-04-22 19:54:13,447 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-22 19:54:13,447 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 in 178 msec 2023-04-22 19:54:13,448 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-22 19:54:13,448 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=98809de8ceb9ac8f3bc773e820fd0e11, REOPEN/MOVE in 498 msec 2023-04-22 19:54:13,451 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-22 19:54:13,452 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-04-22 19:54:13,453 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 943 msec 2023-04-22 19:54:13,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-22 19:54:13,462 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x00cbd61c to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:13,467 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29f78b2c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:13,468 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x33a47c05 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:13,472 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55cbb115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:13,472 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x1aa5eef5 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:13,475 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a71edf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:13,476 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x21265fcb to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:13,481 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24bd3872, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:13,482 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x69f15cb4 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:13,485 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7aba8e46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:13,485 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x0feb770d to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:13,489 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@210d1008, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:13,489 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x54451f11 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:13,492 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c2598b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:13,493 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:13,502 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:13,502 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:13,505 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:13,515 DEBUG [hconnection-0x7648cd86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:13,516 DEBUG [hconnection-0x375264ac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:13,516 DEBUG [hconnection-0x4a3705ee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:13,517 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43766, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:13,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:13,521 DEBUG [hconnection-0x604018af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:13,522 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43768, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:13,521 DEBUG [hconnection-0xad060d3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:13,523 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:13,525 DEBUG [hconnection-0x5913a212-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:13,525 DEBUG [hconnection-0x6bfe7acc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:13,525 DEBUG [hconnection-0x958923d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:13,530 DEBUG [hconnection-0x7a52e68c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:13,532 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43802, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:13,532 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43826, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:13,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:13,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:13,535 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:13,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:13,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:13,536 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:13,536 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:13,536 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43838, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:13,536 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,537 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,537 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43814, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:13,537 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43842, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:13,540 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:13,540 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:13,546 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,546 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:13,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:13,552 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 19:54:13,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:13,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:13,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:13,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:13,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:13,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:13,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:13,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:13,566 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:13,566 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:13,566 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:13,572 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:13,572 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,573 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:13,572 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:13,573 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:13,573 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:13,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:13,576 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48c4cb80[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:13,580 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:13,580 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,586 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:13,586 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:13,586 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:13,586 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,586 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:13,587 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,587 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:13,587 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:13,587 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:13,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:13,590 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:13,590 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:13,590 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:13,590 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:13,590 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:13,592 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,592 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:13,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:13,592 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:13,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:13,593 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:13,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,594 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:13,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,597 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,597 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,597 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,597 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,597 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:13,597 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,597 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:13,597 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:13,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:13,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:13,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:13,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:13,597 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,598 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:13,598 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:13,598 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:13,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:13,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:13,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193313610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193313612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193313612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193313619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43788 deadline: 1682193313621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,696 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 19:54:13,698 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:13,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:13,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:13,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193313723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193313723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193313723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193313727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:13,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:13,730 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:13,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,730 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:13,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43788 deadline: 1682193313728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,730 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:13,730 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:13,731 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,731 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:13,737 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:13,738 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,738 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:13,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:13,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:13,742 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:13,742 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:13,742 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:13,756 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:13,756 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:13,759 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,759 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:13,759 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:13,759 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,759 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:13,759 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:13,760 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,760 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:13,760 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:13,761 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,761 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:13,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:13,762 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,762 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:13,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:13,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:13,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:13,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:13,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3faf5c03[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:13,765 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:13,765 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,766 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,766 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:13,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:13,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:13,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:13,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:13,768 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:13,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:13,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:13,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:13,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:13,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:13,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:13,771 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:13,771 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:13,771 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:13,771 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:13,771 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:13,774 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,774 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:13,774 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:13,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:13,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:13,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:13,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,781 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:13,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:13,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:13,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:13,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:13,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:13,786 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:13,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:13,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:13,786 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:13,786 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:13,887 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:13,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:13,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:13,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:13,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:13,900 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:13,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:13,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:13,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:13,901 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,901 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,902 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,902 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:13,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:13,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:13,902 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,902 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:13,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:13,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:13,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:13,910 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,910 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:13,910 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:13,911 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:13,912 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,912 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:13,913 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,913 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:13,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:13,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ea8ff3c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:13,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:13,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,917 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:13,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:13,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:13,917 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,917 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:13,918 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:13,918 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:13,918 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,918 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:13,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:13,922 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:13,922 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:13,922 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:13,922 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:13,922 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:13,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:13,924 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:13,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:13,926 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:13,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193313927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193313928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193313928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,929 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:13,929 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:13,929 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,929 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,929 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:13,929 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:13,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:13,929 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:13,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:13,930 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:13,930 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,930 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:13,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:13,930 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:13,930 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:13,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:13,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43788 deadline: 1682193313931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:13,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:13,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193313932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,010 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422a99dec55b74745f088f8aa9b8417eaf4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a99dec55b74745f088f8aa9b8417eaf4 2023-04-22 19:54:14,018 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/a7df4132e0ef42778a5a5b319ba66b58 2023-04-22 19:54:14,030 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:14,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:14,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:14,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:14,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:14,040 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:14,040 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,040 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:14,040 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:14,041 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,041 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,043 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,043 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,043 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,043 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:14,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:14,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:14,048 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,048 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:14,049 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:14,049 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,049 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:14,049 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:14,049 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,049 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:14,049 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:14,050 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,050 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:14,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d50e12f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:14,053 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:14,053 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,054 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,055 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:14,055 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:14,055 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:14,056 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:14,056 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,056 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:14,056 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,057 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:14,057 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:14,057 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:14,057 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:14,057 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:14,057 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:14,057 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:14,063 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,063 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,063 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,065 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,069 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,069 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:14,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:14,069 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,069 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:14,069 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:14,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:14,169 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:14,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:14,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:14,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:14,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:14,176 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:14,176 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,176 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:14,176 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:14,184 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,184 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,186 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,186 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,186 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,186 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:14,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:14,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:14,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:14,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:14,210 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:14,211 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,211 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:14,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:14,211 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:14,211 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:14,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:14,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:14,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:14,218 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,218 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,218 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,218 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:14,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43a6a7ce[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:14,220 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:14,220 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,221 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:14,221 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:14,221 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:14,221 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:14,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:14,227 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:14,227 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:14,227 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:14,227 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:14,227 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:14,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:14,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193314232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,234 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,234 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:14,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,234 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193314232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193314233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,236 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,238 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:14,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193314236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:14,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43788 deadline: 1682193314238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,242 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:14,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:14,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,243 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,243 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,243 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:14,243 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:14,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:14,343 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:14,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:14,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:14,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:14,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:14,349 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:14,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:14,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:14,350 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,350 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,351 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,351 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,351 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:14,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:14,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:14,353 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,353 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:14,353 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:14,353 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:14,355 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:14,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6acb4fd[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:14,359 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:14,359 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,361 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,361 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:14,361 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:14,361 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:14,362 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,362 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:14,364 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,364 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:14,364 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,364 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:14,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:14,368 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:14,368 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:14,368 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:14,368 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:14,368 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:14,370 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,370 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,370 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,371 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,372 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:14,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:14,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,376 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:14,376 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:14,378 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:14,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/dbaf3ad1fb564c498b444541cd959f4a 2023-04-22 19:54:14,479 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:14,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:14,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:14,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:14,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:14,484 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:14,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:14,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:14,485 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,485 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,486 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:14,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:14,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:14,489 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:14,489 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,489 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:14,492 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:14,492 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:14,492 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,492 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:14,492 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:14,492 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,492 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:14,492 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:14,492 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,492 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:14,492 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:14,495 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,495 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,495 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,495 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:14,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ff4ea0b[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:14,498 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:14,498 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,500 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:14,500 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:14,500 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:14,500 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,500 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:14,501 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:14,501 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,501 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:14,501 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:14,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:14,506 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:14,506 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:14,506 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:14,506 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:14,506 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:14,508 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,508 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,508 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,509 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,511 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:14,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,516 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,516 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:14,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:14,517 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:14,517 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:14,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,617 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:14,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:14,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:14,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:14,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:14,625 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:14,625 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,625 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:14,625 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:14,625 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,625 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,627 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:14,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:14,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:14,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:14,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:14,628 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:14,628 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,628 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:14,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:14,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:14,632 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,632 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:14,632 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:14,632 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:14,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:14,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:14,633 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:14,634 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,634 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:14,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25873b43[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:14,636 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:14,636 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,637 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:14,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:14,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:14,638 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,638 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:14,643 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,643 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,643 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,643 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:14,646 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:14,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:14,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:14,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:14,646 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:14,648 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,648 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,648 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:14,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:14,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,654 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:14,654 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,654 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:14,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:14,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:14,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193314736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:14,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193314737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:14,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193314739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:14,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193314745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:14,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43788 deadline: 1682193314745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,754 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:14,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:14,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:14,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:14,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:14,767 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:14,767 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,767 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:14,767 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:14,767 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,768 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,769 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,769 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,769 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,769 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:14,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:14,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:14,771 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,771 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:14,772 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:14,772 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:14,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:14,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:14,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:14,790 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,790 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,791 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,791 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:14,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53e960ea[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:14,793 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:14,793 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,794 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,794 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:14,794 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:14,794 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:14,794 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,794 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:14,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:14,796 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,796 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,796 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:14,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:14,798 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:14,798 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:14,798 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:14,799 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:14,799 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:14,800 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,800 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:14,800 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,801 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:14,802 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:14,807 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,807 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,807 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,808 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:14,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:14,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:14,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:14,807 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:14,808 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:14,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:14,808 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:14,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:14,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/ceba049a52a643bd9ea73c60c4bb6a8d 2023-04-22 19:54:14,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/a7df4132e0ef42778a5a5b319ba66b58 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a7df4132e0ef42778a5a5b319ba66b58 2023-04-22 19:54:14,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a7df4132e0ef42778a5a5b319ba66b58, entries=200, sequenceid=18, filesize=31.9 K 2023-04-22 19:54:14,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/dbaf3ad1fb564c498b444541cd959f4a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dbaf3ad1fb564c498b444541cd959f4a 2023-04-22 19:54:14,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dbaf3ad1fb564c498b444541cd959f4a, entries=150, sequenceid=18, filesize=11.6 K 2023-04-22 19:54:14,892 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/ceba049a52a643bd9ea73c60c4bb6a8d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/ceba049a52a643bd9ea73c60c4bb6a8d 2023-04-22 19:54:14,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/ceba049a52a643bd9ea73c60c4bb6a8d, entries=150, sequenceid=18, filesize=11.6 K 2023-04-22 19:54:14,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 98809de8ceb9ac8f3bc773e820fd0e11 in 1356ms, sequenceid=18, compaction requested=false 2023-04-22 19:54:14,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:14,908 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:14,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:14,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:14,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:14,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:14,913 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:14,913 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,913 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:14,913 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:14,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,915 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:14,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:14,916 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,916 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:14,916 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:14,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:14,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:14,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:14,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:14,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:14,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:14,928 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:14,928 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:14,928 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:14,928 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:14,928 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 19:54:14,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:14,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:14,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:14,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:14,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:14,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:15,360 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422432adb4b0d974290a6389ad6d218081b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422432adb4b0d974290a6389ad6d218081b 2023-04-22 19:54:15,369 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/e6292519f0614f908ede7776e062969c 2023-04-22 19:54:15,386 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/a28949d56462493a92d1253c90c0fc21 2023-04-22 19:54:15,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:15,741 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:15,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43788 deadline: 1682193315755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193315755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193315756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193315755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193315757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,808 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/b831ca77571744f9a1868b7027f75bd4 2023-04-22 19:54:15,814 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/e6292519f0614f908ede7776e062969c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/e6292519f0614f908ede7776e062969c 2023-04-22 19:54:15,819 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/e6292519f0614f908ede7776e062969c, entries=150, sequenceid=36, filesize=25.1 K 2023-04-22 19:54:15,820 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/a28949d56462493a92d1253c90c0fc21 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a28949d56462493a92d1253c90c0fc21 2023-04-22 19:54:15,825 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a28949d56462493a92d1253c90c0fc21, entries=150, sequenceid=36, filesize=11.6 K 2023-04-22 19:54:15,826 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/b831ca77571744f9a1868b7027f75bd4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b831ca77571744f9a1868b7027f75bd4 2023-04-22 19:54:15,831 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b831ca77571744f9a1868b7027f75bd4, entries=150, sequenceid=36, filesize=11.6 K 2023-04-22 19:54:15,832 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 98809de8ceb9ac8f3bc773e820fd0e11 in 904ms, sequenceid=36, compaction requested=false 2023-04-22 19:54:15,832 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:15,832 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:15,832 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,836 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:15,836 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:15,837 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,837 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:15,838 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:15,838 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:15,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:15,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30013d36[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:15,840 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:15,840 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,841 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,841 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:15,841 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:15,841 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:15,842 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,842 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:15,843 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:15,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:15,843 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:15,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:15,846 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:15,846 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:15,846 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:15,846 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:15,846 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:15,849 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,849 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:15,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,849 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:15,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,852 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:15,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:15,854 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,856 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:15,856 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:15,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:15,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:15,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:15,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:15,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,857 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:15,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:15,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:15,857 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:15,857 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:15,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:15,861 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 19:54:15,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:15,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:15,863 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:15,863 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:15,863 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:15,863 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:15,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193315877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193315879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193315882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193315882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,890 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042277eec7b5fc2542bca4bc7f7fc26ab8ad to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042277eec7b5fc2542bca4bc7f7fc26ab8ad 2023-04-22 19:54:15,899 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=53, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/d000b63b2b9041729ba40bb28ba127ca 2023-04-22 19:54:15,957 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:15,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:15,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:15,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:15,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:15,963 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:15,963 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:15,964 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:15,964 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:15,964 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,964 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:15,965 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:15,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:15,966 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,966 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:15,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:15,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:15,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:15,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:15,967 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:15,967 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:15,967 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:15,967 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:15,967 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:15,967 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:15,967 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:15,967 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:15,967 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:15,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:15,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:15,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:15,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:15,969 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,969 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:15,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:15,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:15,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188d3d8f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:15,973 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:15,973 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,976 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,976 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:15,976 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:15,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:15,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:15,978 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:15,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:15,981 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:15,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:15,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:15,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:15,981 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:15,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193315984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193315984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:15,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:15,988 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,990 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:15,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:15,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193315989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193315988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:15,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:15,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:15,996 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:15,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:15,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,997 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:15,997 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:15,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:15,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:15,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:15,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:15,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:15,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:15,997 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:15,997 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:16,097 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:16,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:16,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:16,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:16,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:16,101 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:16,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:16,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:16,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:16,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,103 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:16,103 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:16,103 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,103 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:16,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:16,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:16,104 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:16,104 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:16,104 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:16,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:16,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:16,105 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:16,105 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:16,105 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:16,105 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:16,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:16,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:16,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:16,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,110 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:16,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:16,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44c50fe[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:16,112 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:16,112 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,114 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:16,114 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:16,114 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:16,114 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,114 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:16,115 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:16,115 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:16,115 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,115 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:16,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:16,118 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:16,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:16,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:16,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:16,118 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:16,120 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,120 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:16,120 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:16,120 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,122 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,124 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,124 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:16,124 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:16,124 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:16,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:16,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:16,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:16,124 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:16,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,125 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:16,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,125 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:16,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:16,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193316186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193316187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193316194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193316195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,225 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:16,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:16,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:16,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:16,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:16,229 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:16,229 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:16,229 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:16,229 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:16,229 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,229 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,231 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:16,231 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:16,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:16,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:16,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:16,233 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:16,233 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:16,233 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:16,233 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:16,235 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:16,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:16,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cba8ead[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:16,237 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:16,237 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,238 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,238 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:16,238 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:16,238 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:16,238 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,238 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:16,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,240 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:16,240 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:16,240 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:16,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:16,244 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:16,244 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:16,244 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:16,244 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:16,244 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:16,250 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,250 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:16,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,250 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:16,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:16,251 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,253 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,255 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:16,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:16,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:16,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:16,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:16,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:16,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:16,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:16,255 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:16,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:16,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,255 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:16,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:16,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:16,317 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/7f953b5f72ee4b9490d2caf943b0e29c 2023-04-22 19:54:16,329 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/13c53ac0eca4471eba2259c957a6d2b0 2023-04-22 19:54:16,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/d000b63b2b9041729ba40bb28ba127ca as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d000b63b2b9041729ba40bb28ba127ca 2023-04-22 19:54:16,344 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d000b63b2b9041729ba40bb28ba127ca, entries=250, sequenceid=53, filesize=38.7 K 2023-04-22 19:54:16,345 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/7f953b5f72ee4b9490d2caf943b0e29c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/7f953b5f72ee4b9490d2caf943b0e29c 2023-04-22 19:54:16,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/7f953b5f72ee4b9490d2caf943b0e29c, entries=150, sequenceid=53, filesize=11.6 K 2023-04-22 19:54:16,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/13c53ac0eca4471eba2259c957a6d2b0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/13c53ac0eca4471eba2259c957a6d2b0 2023-04-22 19:54:16,355 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:16,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:16,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/13c53ac0eca4471eba2259c957a6d2b0, entries=150, sequenceid=53, filesize=11.6 K 2023-04-22 19:54:16,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 98809de8ceb9ac8f3bc773e820fd0e11 in 502ms, sequenceid=53, compaction requested=true 2023-04-22 19:54:16,362 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:16,362 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:16,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:16,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:16,362 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:16,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:16,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:16,364 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98062 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:16,364 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:16,364 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/A is initiating minor compaction (all files) 2023-04-22 19:54:16,364 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/B is initiating minor compaction (all files) 2023-04-22 19:54:16,364 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/A in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:16,364 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/B in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:16,364 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a7df4132e0ef42778a5a5b319ba66b58, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/e6292519f0614f908ede7776e062969c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d000b63b2b9041729ba40bb28ba127ca] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=95.8 K 2023-04-22 19:54:16,364 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dbaf3ad1fb564c498b444541cd959f4a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a28949d56462493a92d1253c90c0fc21, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/7f953b5f72ee4b9490d2caf943b0e29c] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=34.8 K 2023-04-22 19:54:16,365 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting a7df4132e0ef42778a5a5b319ba66b58, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682193253540 2023-04-22 19:54:16,365 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting dbaf3ad1fb564c498b444541cd959f4a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682193253540 2023-04-22 19:54:16,365 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting e6292519f0614f908ede7776e062969c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682193253571 2023-04-22 19:54:16,366 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a28949d56462493a92d1253c90c0fc21, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682193253571 2023-04-22 19:54:16,366 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting d000b63b2b9041729ba40bb28ba127ca, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1682193255752 2023-04-22 19:54:16,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:16,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:16,366 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 7f953b5f72ee4b9490d2caf943b0e29c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1682193255753 2023-04-22 19:54:16,366 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:16,366 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:16,366 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:16,366 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:16,367 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,367 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,369 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,369 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:16,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:16,369 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,369 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:16,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:16,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:16,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:16,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:16,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:16,371 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:16,371 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:16,371 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:16,376 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:16,376 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:16,376 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:16,376 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:16,376 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-22 19:54:16,376 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:16,376 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:16,376 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:16,376 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:16,377 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:16,377 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:16,385 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#B#compaction#149 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:16,445 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/ee76b953fa5e466895f6b3e5f0c8991d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/ee76b953fa5e466895f6b3e5f0c8991d 2023-04-22 19:54:16,450 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#A#compaction#150 average throughput is 0.32 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:16,459 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/B of 98809de8ceb9ac8f3bc773e820fd0e11 into ee76b953fa5e466895f6b3e5f0c8991d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:16,460 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:16,460 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/B, priority=13, startTime=1682193256362; duration=0sec 2023-04-22 19:54:16,460 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:16,460 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:16,461 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:16,461 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/C is initiating minor compaction (all files) 2023-04-22 19:54:16,461 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/C in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:16,462 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/ceba049a52a643bd9ea73c60c4bb6a8d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b831ca77571744f9a1868b7027f75bd4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/13c53ac0eca4471eba2259c957a6d2b0] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=34.8 K 2023-04-22 19:54:16,462 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ceba049a52a643bd9ea73c60c4bb6a8d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682193253540 2023-04-22 19:54:16,462 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b831ca77571744f9a1868b7027f75bd4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682193253571 2023-04-22 19:54:16,463 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 13c53ac0eca4471eba2259c957a6d2b0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1682193255753 2023-04-22 19:54:16,473 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#C#compaction#152 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:16,493 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:16,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:16,520 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422172a736b0524433cad5fb26a18ec2fab to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422172a736b0524433cad5fb26a18ec2fab 2023-04-22 19:54:16,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193316524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193316524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193316526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193316526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193316630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193316630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193316634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193316637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193316834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193316834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193316838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:16,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193316840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:16,924 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/944a65c174704c22aa8e6bc4746c55e4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/944a65c174704c22aa8e6bc4746c55e4 2023-04-22 19:54:16,930 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/C of 98809de8ceb9ac8f3bc773e820fd0e11 into 944a65c174704c22aa8e6bc4746c55e4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:16,930 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:16,930 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/C, priority=13, startTime=1682193256362; duration=0sec 2023-04-22 19:54:16,930 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:16,959 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/cf6fe307f0d748e387d2003b67313de0 2023-04-22 19:54:16,965 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/8c9fbfea5829465c9cbf2c34f7b4c8cf as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/8c9fbfea5829465c9cbf2c34f7b4c8cf 2023-04-22 19:54:16,972 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/A of 98809de8ceb9ac8f3bc773e820fd0e11 into 8c9fbfea5829465c9cbf2c34f7b4c8cf(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:16,972 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:16,972 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/A, priority=13, startTime=1682193256362; duration=0sec 2023-04-22 19:54:16,973 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:17,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193317137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193317137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193317142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193317142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,376 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/18a362f4211f4fd0816867f660002418 2023-04-22 19:54:17,408 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/4ea24463c1e5450e922e2c5218b932f4 2023-04-22 19:54:17,413 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/cf6fe307f0d748e387d2003b67313de0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/cf6fe307f0d748e387d2003b67313de0 2023-04-22 19:54:17,419 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/cf6fe307f0d748e387d2003b67313de0, entries=150, sequenceid=69, filesize=25.1 K 2023-04-22 19:54:17,421 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/18a362f4211f4fd0816867f660002418 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/18a362f4211f4fd0816867f660002418 2023-04-22 19:54:17,426 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/18a362f4211f4fd0816867f660002418, entries=150, sequenceid=69, filesize=11.6 K 2023-04-22 19:54:17,428 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/4ea24463c1e5450e922e2c5218b932f4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4ea24463c1e5450e922e2c5218b932f4 2023-04-22 19:54:17,434 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4ea24463c1e5450e922e2c5218b932f4, entries=150, sequenceid=69, filesize=11.6 K 2023-04-22 19:54:17,435 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 98809de8ceb9ac8f3bc773e820fd0e11 in 1058ms, sequenceid=69, compaction requested=false 2023-04-22 19:54:17,435 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:17,435 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:17,435 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:17,435 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:17,435 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,435 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:17,435 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:17,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:17,438 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:17,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:17,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25ebe881[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:17,440 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:17,440 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:17,442 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:17,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:17,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:17,442 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:17,443 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:17,443 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:17,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:17,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:17,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:17,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:17,444 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:17,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:17,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:17,447 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:17,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:17,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:17,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:17,448 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:17,449 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:17,449 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:17,449 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:17,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:17,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:17,450 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:17,452 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,456 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,456 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:17,456 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:17,456 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,456 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:17,456 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:17,456 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:17,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:17,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:17,456 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:17,456 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:17,456 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:17,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:17,456 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:17,456 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:17,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:17,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:17,556 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:17,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:17,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:17,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:17,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:17,560 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:17,560 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:17,561 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:17,561 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:17,561 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:17,561 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:17,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,562 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:17,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:17,563 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,563 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:17,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:17,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:17,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:17,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:17,564 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:17,564 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:17,564 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:17,564 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:17,564 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:17,564 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:17,564 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:17,565 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-22 19:54:17,565 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:17,565 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:17,565 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:17,565 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:17,565 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:17,565 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:17,581 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304225ebced96946f450bacc4edffe4c12596 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225ebced96946f450bacc4edffe4c12596 2023-04-22 19:54:17,604 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=89, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/7416387d57ca4ecb9f0f32489f72f912 2023-04-22 19:54:17,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:17,647 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:17,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193317655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193317655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193317656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193317657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193317760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193317760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193317760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193317760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43788 deadline: 1682193317769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,772 DEBUG [Thread-1000] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:54:17,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193317964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193317965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193317966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:17,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:17,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193317966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,027 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/828ec8a108b049f081041e89dd5f10ac 2023-04-22 19:54:18,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193318267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193318268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193318268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193318271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,448 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/fba0a0f722a64b5098c6bc293c8f0f6b 2023-04-22 19:54:18,453 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/7416387d57ca4ecb9f0f32489f72f912 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/7416387d57ca4ecb9f0f32489f72f912 2023-04-22 19:54:18,457 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/7416387d57ca4ecb9f0f32489f72f912, entries=150, sequenceid=89, filesize=25.1 K 2023-04-22 19:54:18,459 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/828ec8a108b049f081041e89dd5f10ac as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/828ec8a108b049f081041e89dd5f10ac 2023-04-22 19:54:18,463 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/828ec8a108b049f081041e89dd5f10ac, entries=150, sequenceid=89, filesize=11.6 K 2023-04-22 19:54:18,464 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/fba0a0f722a64b5098c6bc293c8f0f6b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/fba0a0f722a64b5098c6bc293c8f0f6b 2023-04-22 19:54:18,468 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/fba0a0f722a64b5098c6bc293c8f0f6b, entries=150, sequenceid=89, filesize=11.6 K 2023-04-22 19:54:18,469 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 98809de8ceb9ac8f3bc773e820fd0e11 in 904ms, sequenceid=89, compaction requested=true 2023-04-22 19:54:18,469 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:18,469 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:18,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:18,471 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:18,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:18,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:18,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:18,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547bd8f6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:18,473 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:18,473 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:18,474 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:18,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:18,475 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,475 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:18,477 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,477 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:18,477 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:18,477 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:18,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:18,480 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:18,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:18,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:18,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:18,480 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:18,482 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:18,482 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:18,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:18,482 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:18,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:18,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:18,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:18,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:18,483 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:18,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:18,485 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:18,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:18,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:18,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:18,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:18,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:18,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:18,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:18,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:18,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:18,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:18,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:18,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:18,488 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:18,488 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:18,588 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:18,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:18,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:18,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:18,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:18,592 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:18,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:18,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:18,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:18,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:18,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:18,593 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,593 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:18,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:18,594 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,594 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:18,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:18,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:18,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:18,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:18,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:18,595 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:18,595 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:18,595 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:18,595 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:18,595 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:18,595 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:18,595 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:18,595 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 19:54:18,596 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:18,596 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:18,596 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:18,596 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:18,596 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:18,596 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:18,612 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 19:54:18,631 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422c702b8f53efa4b80a72672b93e631413 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c702b8f53efa4b80a72672b93e631413 2023-04-22 19:54:18,657 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=104, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/151d60b73acd4934a7431131198d9679 2023-04-22 19:54:18,670 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/19731fb4f43d478fa6ce3bd69b759a2c 2023-04-22 19:54:18,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:18,774 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:18,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193318792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193318793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193318794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193318794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193318896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193318899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193318899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:18,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:18,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193318899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,082 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/25ef82cdde3743d1ab6073fa0480b256 2023-04-22 19:54:19,087 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/151d60b73acd4934a7431131198d9679 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/151d60b73acd4934a7431131198d9679 2023-04-22 19:54:19,091 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/151d60b73acd4934a7431131198d9679, entries=150, sequenceid=104, filesize=25.1 K 2023-04-22 19:54:19,092 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/19731fb4f43d478fa6ce3bd69b759a2c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/19731fb4f43d478fa6ce3bd69b759a2c 2023-04-22 19:54:19,098 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/19731fb4f43d478fa6ce3bd69b759a2c, entries=150, sequenceid=104, filesize=11.6 K 2023-04-22 19:54:19,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,099 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/25ef82cdde3743d1ab6073fa0480b256 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/25ef82cdde3743d1ab6073fa0480b256 2023-04-22 19:54:19,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193319099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193319102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193319102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193319103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,104 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/25ef82cdde3743d1ab6073fa0480b256, entries=150, sequenceid=104, filesize=11.6 K 2023-04-22 19:54:19,105 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 98809de8ceb9ac8f3bc773e820fd0e11 in 510ms, sequenceid=104, compaction requested=true 2023-04-22 19:54:19,105 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:19,105 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:19,111 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:19,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:19,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d6b927[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:19,113 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:19,113 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:19,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:19,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:19,114 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,115 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:19,116 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:19,120 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:19,120 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:19,120 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:19,120 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:19,120 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:19,121 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:19,121 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:19,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:19,121 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:19,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:19,122 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,128 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:19,128 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:19,128 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,128 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:19,128 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:19,128 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,128 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:19,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:19,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:19,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:19,128 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:19,128 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:19,128 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:19,129 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:19,129 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:19,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:19,229 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:19,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:19,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:19,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:19,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:19,232 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:19,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:19,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:19,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:19,233 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:19,233 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:19,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,234 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:19,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:19,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:19,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:19,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:19,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:19,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:19,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:19,236 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 19:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:19,237 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:19,251 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422d8ed4b72b75f4336953d35c7bf5d3776 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d8ed4b72b75f4336953d35c7bf5d3776 2023-04-22 19:54:19,261 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=122, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/a1f73b43770f478f8259339b5e4aa4cd 2023-04-22 19:54:19,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:19,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:19,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193319414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193319415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193319416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193319417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193319521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193319521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193319521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193319521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,678 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/cd17d3e499b44bb1ae370569c2a19bc6 2023-04-22 19:54:19,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193319724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193319724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193319725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:19,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:19,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193319727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193320026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193320027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193320028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193320031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,090 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/7885e54aaf894a8ca4d57011365cfccf 2023-04-22 19:54:20,096 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/a1f73b43770f478f8259339b5e4aa4cd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a1f73b43770f478f8259339b5e4aa4cd 2023-04-22 19:54:20,101 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a1f73b43770f478f8259339b5e4aa4cd, entries=150, sequenceid=122, filesize=25.1 K 2023-04-22 19:54:20,102 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/cd17d3e499b44bb1ae370569c2a19bc6 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/cd17d3e499b44bb1ae370569c2a19bc6 2023-04-22 19:54:20,106 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/cd17d3e499b44bb1ae370569c2a19bc6, entries=150, sequenceid=122, filesize=11.6 K 2023-04-22 19:54:20,107 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/7885e54aaf894a8ca4d57011365cfccf as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/7885e54aaf894a8ca4d57011365cfccf 2023-04-22 19:54:20,118 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/7885e54aaf894a8ca4d57011365cfccf, entries=150, sequenceid=122, filesize=11.6 K 2023-04-22 19:54:20,119 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 98809de8ceb9ac8f3bc773e820fd0e11 in 883ms, sequenceid=122, compaction requested=true 2023-04-22 19:54:20,119 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:20,119 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:20,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:20,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:20,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:20,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:20,122 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,122 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:20,122 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,122 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad19636[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:20,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:20,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:20,125 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,125 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:20,128 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:20,128 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:20,128 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,128 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:20,130 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:20,130 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:20,130 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:20,130 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:20,130 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:20,132 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,132 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,132 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:20,133 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,134 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:20,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:20,140 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:20,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:20,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:20,140 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:20,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:20,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:20,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,140 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:20,140 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:20,140 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:20,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:20,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:20,240 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:20,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:20,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:20,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:20,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:20,244 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:20,244 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:20,244 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:20,244 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:20,244 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,245 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,246 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,246 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:20,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:20,246 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,246 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:20,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:20,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:20,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:20,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:20,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:20,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:20,257 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:20,258 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:20,258 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 19:54:20,258 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:20,258 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:20,258 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:20,258 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:20,258 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:20,258 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:20,281 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422178dc211f92148fd8abde53924118e71 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422178dc211f92148fd8abde53924118e71 2023-04-22 19:54:20,296 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=136, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/f3adc7f225af470281601abe0df8d30c 2023-04-22 19:54:20,317 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/a06497dace1b4dc692b562c8d5b04b72 2023-04-22 19:54:20,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:20,534 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:20,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193320546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193320549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193320550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193320550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193320651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193320651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193320654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193320654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,728 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/f08b6d0213dc48edb4115d28df50bc89 2023-04-22 19:54:20,732 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/f3adc7f225af470281601abe0df8d30c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/f3adc7f225af470281601abe0df8d30c 2023-04-22 19:54:20,736 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/f3adc7f225af470281601abe0df8d30c, entries=150, sequenceid=136, filesize=25.3 K 2023-04-22 19:54:20,737 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/a06497dace1b4dc692b562c8d5b04b72 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a06497dace1b4dc692b562c8d5b04b72 2023-04-22 19:54:20,741 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a06497dace1b4dc692b562c8d5b04b72, entries=150, sequenceid=136, filesize=11.8 K 2023-04-22 19:54:20,742 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/f08b6d0213dc48edb4115d28df50bc89 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f08b6d0213dc48edb4115d28df50bc89 2023-04-22 19:54:20,746 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f08b6d0213dc48edb4115d28df50bc89, entries=150, sequenceid=136, filesize=11.8 K 2023-04-22 19:54:20,747 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 98809de8ceb9ac8f3bc773e820fd0e11 in 489ms, sequenceid=136, compaction requested=true 2023-04-22 19:54:20,748 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:20,748 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:20,749 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:20,750 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,750 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:20,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:20,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@84c60d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:20,751 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:20,751 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,753 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,754 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:20,754 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:20,754 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:20,754 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,754 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:20,755 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:20,755 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,755 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:20,755 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:20,758 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:20,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:20,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:20,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:20,758 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:20,759 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,759 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:20,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,759 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:20,760 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:20,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:20,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:20,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:20,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,766 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:20,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:20,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:20,767 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:20,767 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:20,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:20,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 19:54:20,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:20,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:20,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:20,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:20,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:20,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:20,867 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:20,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:20,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:20,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:20,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:20,873 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:20,873 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:20,873 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:20,873 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:20,874 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,874 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,875 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:20,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:20,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:20,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193320873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:20,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193320874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:20,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:20,877 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:20,877 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:20,877 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:20,877 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:20,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193320876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193320876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,881 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:20,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:20,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ab21283[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:20,883 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:20,883 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,884 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,884 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:20,885 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:20,885 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:20,885 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,885 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:20,886 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:20,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:20,889 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:20,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:20,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:20,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:20,889 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:20,891 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,891 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:20,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,891 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:20,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:20,891 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,893 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,895 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,895 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:20,896 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:20,896 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:20,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:20,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:20,896 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:20,896 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:20,896 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,896 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:20,896 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:20,896 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:20,896 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:20,896 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:20,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193320977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193320977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193320981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:20,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193320981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:20,996 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:20,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:20,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:21,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:21,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:21,000 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:21,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:21,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:21,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:21,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:21,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:21,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:21,003 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:21,003 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,003 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:21,004 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,004 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:21,004 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:21,004 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:21,005 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,005 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,006 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,006 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:21,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d2a9d73[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:21,007 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:21,007 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,008 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:21,008 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,009 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:21,011 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:21,011 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,011 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:21,011 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:21,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:21,014 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:21,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:21,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:21,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:21,014 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:21,016 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,016 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,016 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,016 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,020 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,020 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,020 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,020 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:21,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:21,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:21,020 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,020 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,020 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,020 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:21,020 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,021 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:21,021 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,021 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,121 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:21,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:21,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:21,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:21,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:21,124 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:21,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:21,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:21,125 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,125 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,126 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,126 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,127 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,127 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:21,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:21,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:21,128 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:21,128 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,128 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:21,128 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:21,128 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:21,129 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,129 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:21,129 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:21,129 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:21,134 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,134 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,134 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,134 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b3cdac6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:21,135 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:21,135 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,137 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:21,137 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:21,137 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:21,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:21,138 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:21,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:21,140 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:21,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:21,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:21,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:21,141 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:21,143 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,143 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,143 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,143 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,145 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,147 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,147 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,147 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,147 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,147 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,147 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:21,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:21,147 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,147 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,147 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:21,147 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,148 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:21,148 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:21,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193321181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193321181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193321182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193321183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,248 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:21,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:21,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:21,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:21,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:21,253 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:21,253 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,254 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:21,254 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:21,254 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,254 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,255 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,255 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,256 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,256 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:21,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:21,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:21,257 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,257 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:21,257 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:21,257 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:21,258 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,258 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:21,258 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:21,259 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:21,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5daecf02[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:21,261 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:21,261 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,263 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:21,263 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:21,263 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:21,265 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:21,267 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:21,267 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:21,268 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:21,268 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:21,268 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:21,269 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,269 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,269 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,270 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,272 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,274 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304229134995d4eb2425ea43dcbd121fc4f08 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304229134995d4eb2425ea43dcbd121fc4f08 2023-04-22 19:54:21,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:21,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:21,275 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,276 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:21,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,276 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:21,376 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:21,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:21,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:21,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:21,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:21,380 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:21,380 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,380 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:21,380 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:21,380 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,380 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,382 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,382 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,383 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,383 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:21,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:21,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:21,384 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:21,384 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,384 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:21,384 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:21,384 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:21,384 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,384 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:21,384 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:21,385 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:21,386 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,386 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,386 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,386 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:21,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4792e83e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:21,388 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:21,388 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,389 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:21,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:21,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:21,389 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,389 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:21,391 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:21,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:21,393 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:21,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:21,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:21,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:21,393 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:21,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,396 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,398 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,400 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,400 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,400 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,400 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,400 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,400 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,400 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,400 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:21,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:21,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,400 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:21,400 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,401 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:21,401 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:21,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193321485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193321485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193321487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193321489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,501 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:21,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:21,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:21,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:21,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:21,504 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:21,504 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,504 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:21,504 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:21,505 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,505 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,506 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,506 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,506 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,506 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:21,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:21,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:21,508 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,508 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:21,508 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:21,508 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,510 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:21,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e0a24ba[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:21,512 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:21,512 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,514 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:21,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:21,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:21,515 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,515 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:21,518 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:21,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:21,521 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:21,521 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:21,521 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:21,521 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:21,521 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:21,523 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,523 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,523 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,523 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,525 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,527 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,527 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,527 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:21,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,527 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,527 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,527 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,527 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,527 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,527 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,528 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,527 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:21,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:21,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,528 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:21,628 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:21,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:21,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:21,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:21,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:21,631 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:21,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:21,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:21,632 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,632 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,634 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,634 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,634 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,634 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:21,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:21,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:21,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:21,636 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,636 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:21,636 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:21,636 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:21,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37d2757[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:21,639 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:21,639 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,640 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:21,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:21,642 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:21,645 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:21,645 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:21,645 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:21,645 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:21,645 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:21,648 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,648 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,648 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,650 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,652 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,652 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,652 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,652 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,652 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:21,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:21,652 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,652 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,653 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:21,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,653 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:21,686 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=156, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/3036225b621a4994b947ce0369f3728e 2023-04-22 19:54:21,753 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:21,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:21,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:21,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:21,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:21,757 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:21,757 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,757 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:21,757 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:21,757 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,757 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,759 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:21,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:21,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:21,764 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,764 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:21,764 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:21,764 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:21,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:21,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,767 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,767 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28803ee5[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:21,769 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:21,769 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:21,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:21,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:21,772 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,772 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:21,773 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:21,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:21,776 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:21,776 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:21,776 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:21,776 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:21,776 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:21,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,778 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,780 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43788 deadline: 1682193321777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,783 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:21,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:21,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:21,783 DEBUG [Thread-1000] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:54:21,784 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:21,784 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:21,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,884 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:21,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:21,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:21,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:21,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:21,888 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:21,888 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,888 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:21,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:21,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,892 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,892 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,892 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,893 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:21,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:21,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:21,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:21,894 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:21,894 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:21,894 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:21,894 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:21,894 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:21,895 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,895 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:21,895 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:21,895 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:21,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:21,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:21,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:21,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:21,896 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,897 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7883e0d3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:21,899 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:21,899 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:21,901 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,901 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:21,903 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:21,903 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,903 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,903 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:21,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:21,907 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:21,908 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:21,908 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:21,908 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:21,908 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:21,913 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,914 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,914 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:21,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,917 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,918 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,921 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:21,921 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:21,922 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,922 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,922 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:21,922 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,922 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,922 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:21,922 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:21,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:21,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:21,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:21,922 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:21,922 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:21,922 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:21,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:21,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193321990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193321992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193321992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:21,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:21,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193321994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,023 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:22,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:22,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:22,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:22,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:22,028 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:22,028 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,028 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:22,028 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:22,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,031 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:22,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:22,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:22,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:22,034 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,034 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:22,034 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:22,034 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:22,037 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,037 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,037 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,037 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e3348d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:22,044 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:22,044 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,046 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:22,046 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,046 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:22,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:22,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:22,049 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:22,052 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:22,052 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:22,053 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:22,053 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:22,053 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:22,055 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,055 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,056 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,059 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,061 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,065 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,065 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,066 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,066 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,066 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,066 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,066 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,066 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,066 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:22,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:22,066 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:22,066 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:22,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,103 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/49162f48167e4a9e89dafcd9698b5ccb 2023-04-22 19:54:22,166 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:22,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:22,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:22,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:22,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:22,172 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:22,172 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,172 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:22,172 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:22,172 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,173 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,174 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,174 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:22,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:22,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:22,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:22,176 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,176 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:22,176 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:22,177 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,177 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:22,177 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:22,177 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,177 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:22,177 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:22,178 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:22,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dceeab9[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:22,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:22,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,182 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:22,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:22,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:22,182 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,182 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:22,185 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:22,185 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,185 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:22,185 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:22,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:22,192 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:22,192 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:22,192 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:22,192 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:22,192 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:22,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,195 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,196 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:22,199 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:22,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,200 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:22,200 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:22,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,300 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:22,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:22,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:22,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:22,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:22,304 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:22,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:22,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:22,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,305 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,306 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,306 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,306 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,306 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:22,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:22,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:22,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:22,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:22,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:22,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:22,308 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,308 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:22,308 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:22,309 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,309 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:22,309 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:22,309 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,309 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:22,309 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:22,315 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,315 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:22,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@741080f3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:22,318 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:22,318 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,319 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:22,319 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:22,319 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:22,319 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:22,321 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:22,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:22,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:22,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:22,324 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:22,324 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:22,324 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:22,324 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:22,324 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:22,326 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,326 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,326 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,327 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,332 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,344 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:22,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,344 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,344 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,344 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,344 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,344 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,344 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:22,344 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:22,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,444 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:22,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:22,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:22,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:22,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:22,448 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:22,448 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,448 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:22,448 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:22,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,450 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,450 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,451 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,451 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:22,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:22,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:22,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:22,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:22,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:22,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:22,452 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,452 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:22,452 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:22,453 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,453 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:22,453 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:22,453 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,453 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:22,453 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:22,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,455 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,455 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:22,460 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:22,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13f263da[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:22,462 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:22,462 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,464 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:22,465 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,465 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:22,466 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:22,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:22,469 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:22,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:22,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:22,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:22,469 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:22,471 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,471 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,471 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,473 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,478 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,478 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,479 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:22,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,480 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:22,480 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:22,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/c284536ba6d44e40ac0ae3a365c23a91 2023-04-22 19:54:22,534 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/3036225b621a4994b947ce0369f3728e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/3036225b621a4994b947ce0369f3728e 2023-04-22 19:54:22,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/3036225b621a4994b947ce0369f3728e, entries=200, sequenceid=156, filesize=32.1 K 2023-04-22 19:54:22,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/49162f48167e4a9e89dafcd9698b5ccb as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/49162f48167e4a9e89dafcd9698b5ccb 2023-04-22 19:54:22,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/49162f48167e4a9e89dafcd9698b5ccb, entries=150, sequenceid=156, filesize=11.8 K 2023-04-22 19:54:22,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/c284536ba6d44e40ac0ae3a365c23a91 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/c284536ba6d44e40ac0ae3a365c23a91 2023-04-22 19:54:22,551 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/c284536ba6d44e40ac0ae3a365c23a91, entries=150, sequenceid=156, filesize=11.8 K 2023-04-22 19:54:22,552 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 98809de8ceb9ac8f3bc773e820fd0e11 in 1694ms, sequenceid=156, compaction requested=true 2023-04-22 19:54:22,552 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:22,552 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:22,552 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:22,552 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 19:54:22,552 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-22 19:54:22,553 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 19:54:22,558 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 187615 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 19:54:22,558 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/A is initiating minor compaction (all files) 2023-04-22 19:54:22,559 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/A in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,559 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/8c9fbfea5829465c9cbf2c34f7b4c8cf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/cf6fe307f0d748e387d2003b67313de0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/7416387d57ca4ecb9f0f32489f72f912, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/151d60b73acd4934a7431131198d9679, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a1f73b43770f478f8259339b5e4aa4cd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/f3adc7f225af470281601abe0df8d30c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/3036225b621a4994b947ce0369f3728e] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=183.2 K 2023-04-22 19:54:22,559 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 8c9fbfea5829465c9cbf2c34f7b4c8cf, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1682193255753 2023-04-22 19:54:22,560 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 19:54:22,560 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/B is initiating minor compaction (all files) 2023-04-22 19:54:22,560 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/B in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,560 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/ee76b953fa5e466895f6b3e5f0c8991d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/18a362f4211f4fd0816867f660002418, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/828ec8a108b049f081041e89dd5f10ac, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/19731fb4f43d478fa6ce3bd69b759a2c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/cd17d3e499b44bb1ae370569c2a19bc6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a06497dace1b4dc692b562c8d5b04b72, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/49162f48167e4a9e89dafcd9698b5ccb] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=81.6 K 2023-04-22 19:54:22,560 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting cf6fe307f0d748e387d2003b67313de0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682193255867 2023-04-22 19:54:22,561 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ee76b953fa5e466895f6b3e5f0c8991d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1682193255753 2023-04-22 19:54:22,561 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 7416387d57ca4ecb9f0f32489f72f912, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1682193256515 2023-04-22 19:54:22,561 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 18a362f4211f4fd0816867f660002418, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682193255867 2023-04-22 19:54:22,561 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 151d60b73acd4934a7431131198d9679, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1682193257651 2023-04-22 19:54:22,562 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 828ec8a108b049f081041e89dd5f10ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1682193256515 2023-04-22 19:54:22,562 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting a1f73b43770f478f8259339b5e4aa4cd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1682193258774 2023-04-22 19:54:22,562 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 19731fb4f43d478fa6ce3bd69b759a2c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1682193257651 2023-04-22 19:54:22,562 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting f3adc7f225af470281601abe0df8d30c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682193259410 2023-04-22 19:54:22,562 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting cd17d3e499b44bb1ae370569c2a19bc6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1682193258774 2023-04-22 19:54:22,563 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 3036225b621a4994b947ce0369f3728e, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682193260549 2023-04-22 19:54:22,563 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a06497dace1b4dc692b562c8d5b04b72, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682193259410 2023-04-22 19:54:22,564 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 49162f48167e4a9e89dafcd9698b5ccb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682193260549 2023-04-22 19:54:22,580 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:22,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:22,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:22,594 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#B#compaction#171 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:22,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:22,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:22,594 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:22,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:22,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:22,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,597 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:22,600 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#A#compaction#170 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:22,601 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,601 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,601 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:22,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:22,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:22,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:22,609 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:22,609 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:22,609 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,609 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:22,609 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 19:54:22,609 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:22,609 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:22,610 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:22,610 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:22,610 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:22,610 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:22,629 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/00a514c9d55d40579f61642f04a326f3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/00a514c9d55d40579f61642f04a326f3 2023-04-22 19:54:22,635 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042256332e873add4414b8c335c5e82e09d2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042256332e873add4414b8c335c5e82e09d2 2023-04-22 19:54:22,639 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/B of 98809de8ceb9ac8f3bc773e820fd0e11 into 00a514c9d55d40579f61642f04a326f3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:22,639 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:22,639 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/B, priority=9, startTime=1682193262552; duration=0sec 2023-04-22 19:54:22,640 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:22,640 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 19:54:22,643 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 19:54:22,643 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/C is initiating minor compaction (all files) 2023-04-22 19:54:22,643 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/C in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,643 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/944a65c174704c22aa8e6bc4746c55e4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4ea24463c1e5450e922e2c5218b932f4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/fba0a0f722a64b5098c6bc293c8f0f6b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/25ef82cdde3743d1ab6073fa0480b256, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/7885e54aaf894a8ca4d57011365cfccf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f08b6d0213dc48edb4115d28df50bc89, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/c284536ba6d44e40ac0ae3a365c23a91] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=81.6 K 2023-04-22 19:54:22,644 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 944a65c174704c22aa8e6bc4746c55e4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1682193255753 2023-04-22 19:54:22,644 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 4ea24463c1e5450e922e2c5218b932f4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682193255867 2023-04-22 19:54:22,645 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting fba0a0f722a64b5098c6bc293c8f0f6b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1682193256515 2023-04-22 19:54:22,645 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 25ef82cdde3743d1ab6073fa0480b256, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1682193257651 2023-04-22 19:54:22,646 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 7885e54aaf894a8ca4d57011365cfccf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1682193258774 2023-04-22 19:54:22,646 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f08b6d0213dc48edb4115d28df50bc89, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682193259410 2023-04-22 19:54:22,647 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting c284536ba6d44e40ac0ae3a365c23a91, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682193260549 2023-04-22 19:54:22,658 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/bb1950696c92409ca86d3fdb83f8cff5 2023-04-22 19:54:22,671 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#C#compaction#173 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:22,687 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/b27a4fc977cd43449cdbd801bbf27044 2023-04-22 19:54:22,704 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/e862bb4f25f147288aa7232f884cf4b8 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/e862bb4f25f147288aa7232f884cf4b8 2023-04-22 19:54:22,707 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/b27e163339434456b291b30f86a1f8cc 2023-04-22 19:54:22,712 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/C of 98809de8ceb9ac8f3bc773e820fd0e11 into e862bb4f25f147288aa7232f884cf4b8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:22,712 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:22,712 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/C, priority=9, startTime=1682193262552; duration=0sec 2023-04-22 19:54:22,712 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:22,715 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/bb1950696c92409ca86d3fdb83f8cff5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/bb1950696c92409ca86d3fdb83f8cff5 2023-04-22 19:54:22,721 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/bb1950696c92409ca86d3fdb83f8cff5, entries=100, sequenceid=169, filesize=18.5 K 2023-04-22 19:54:22,721 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/b27a4fc977cd43449cdbd801bbf27044 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/b27a4fc977cd43449cdbd801bbf27044 2023-04-22 19:54:22,726 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/b27a4fc977cd43449cdbd801bbf27044, entries=100, sequenceid=169, filesize=9.4 K 2023-04-22 19:54:22,727 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/b27e163339434456b291b30f86a1f8cc as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b27e163339434456b291b30f86a1f8cc 2023-04-22 19:54:22,738 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b27e163339434456b291b30f86a1f8cc, entries=100, sequenceid=169, filesize=9.4 K 2023-04-22 19:54:22,739 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 98809de8ceb9ac8f3bc773e820fd0e11 in 130ms, sequenceid=169, compaction requested=false 2023-04-22 19:54:22,739 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:22,739 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,740 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:22,740 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:22,745 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,745 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,745 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,745 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d9b6a75[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:22,747 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:22,747 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,749 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:22,749 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,749 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:22,750 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,751 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,751 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:22,754 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:22,754 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:22,754 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:22,754 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:22,754 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:22,757 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,757 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,757 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,762 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,762 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,762 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,762 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:22,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,762 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:22,763 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:22,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,763 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:22,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,863 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:22,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:22,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:22,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:22,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:22,867 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:22,867 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,867 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:22,867 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:22,868 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,868 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,869 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,869 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,869 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:22,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:22,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:22,871 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,871 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:22,871 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:22,871 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:22,873 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,873 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,873 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,873 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@172ac45d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:22,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:22,876 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,878 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:22,878 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:22,878 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:22,878 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,878 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:22,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:22,879 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:22,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:22,882 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:22,882 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:22,883 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:22,883 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:22,883 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:22,884 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,884 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:22,885 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:22,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,885 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:22,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,887 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,893 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:22,893 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:22,893 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,893 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:22,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:22,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:22,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:22,893 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:22,893 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:22,894 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:22,894 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,894 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:22,894 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,893 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:22,894 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:22,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:22,994 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:22,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:22,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:22,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:22,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:22,998 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:22,999 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:22,999 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:22,999 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:22,999 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:22,999 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,003 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,003 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:23,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:23,012 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,012 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:23,012 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:23,012 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:23,014 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,014 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,015 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,015 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:23,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@674dd953[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:23,017 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:23,017 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,018 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:23,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:23,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:23,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:23,019 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,020 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,020 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:23,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:23,023 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:23,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:23,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:23,024 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:23,024 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:23,027 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,027 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,027 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,028 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:23,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:23,033 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,033 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,033 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:23,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:23,033 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:23,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:23,035 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:54:23,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:23,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:23,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:23,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:23,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:23,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:23,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193323074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193323075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682193263041, max timestamp=1682193263080Num uniques 859; 2023-04-22 19:54:23,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682193263041, max timestamp=1682193263080Num uniques 859; 2023-04-22 19:54:23,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682193263041, max timestamp=1682193263080Num uniques 859; 2023-04-22 19:54:23,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-22 19:54:23,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-22 19:54:23,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-22 19:54:23,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193323091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193323099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,133 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:23,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:23,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:23,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:23,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:23,137 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:23,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:23,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:23,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,138 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:23,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:23,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:23,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:23,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:23,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,141 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:23,141 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:23,142 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,142 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,142 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67a19ad3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:23,144 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:23,144 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,146 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:23,147 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,147 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:23,148 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:23,151 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:23,151 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:23,151 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:23,151 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:23,151 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:23,153 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,153 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,153 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,157 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,166 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,166 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,166 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:23,166 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:23,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:23,166 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,166 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,166 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,166 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,166 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:23,167 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:23,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,166 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,167 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193323183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193323185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193323200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193323203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,267 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:23,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:23,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:23,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:23,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:23,270 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:23,270 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,270 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:23,270 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:23,271 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,271 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,273 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,273 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,274 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,274 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:23,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:23,276 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,276 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:23,276 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:23,276 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:23,278 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,279 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f8c90[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:23,280 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:23,280 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,282 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,282 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:23,282 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:23,282 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:23,282 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,282 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:23,284 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:23,288 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:23,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:23,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:23,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:23,288 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:23,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,291 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,293 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:23,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:23,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,296 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:23,296 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:23,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193323385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193323389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,396 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:23,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:23,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:23,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:23,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:23,400 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:23,400 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,400 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:23,401 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:23,401 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,401 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,402 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,402 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,403 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,403 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:23,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:23,405 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,405 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:23,406 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:23,406 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:23,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193323404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,412 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193323409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,413 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,413 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:23,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c6b8c0[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:23,415 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:23,415 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:23,416 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,416 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:23,418 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:23,418 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,418 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,418 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:23,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:23,420 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:23,420 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:23,420 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:23,420 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:23,420 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:23,422 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,422 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,422 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,422 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,424 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,427 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,427 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,427 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,427 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:23,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,427 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,428 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:23,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,428 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:23,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:23,457 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422924b57fd084c4a9087677b42e022a1e9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422924b57fd084c4a9087677b42e022a1e9 2023-04-22 19:54:23,462 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/d743495e03d9449ca5feb5145299983e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d743495e03d9449ca5feb5145299983e 2023-04-22 19:54:23,464 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=184, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/d41d54d42c854164a073f895847b7c14 2023-04-22 19:54:23,468 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/A of 98809de8ceb9ac8f3bc773e820fd0e11 into d743495e03d9449ca5feb5145299983e(size=25.6 K), total size for store is 44.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:23,468 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:23,468 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/A, priority=9, startTime=1682193262552; duration=0sec 2023-04-22 19:54:23,468 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:23,528 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:23,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:23,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:23,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:23,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:23,531 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:23,531 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,531 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:23,531 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:23,532 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,532 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,533 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,533 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,533 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,533 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:23,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:23,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:23,535 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,535 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:23,535 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:23,535 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:23,540 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:23,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49c876ca[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:23,542 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:23,542 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,543 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,543 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,543 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:23,543 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:23,543 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:23,543 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:23,546 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:23,546 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,546 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,546 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:23,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:23,548 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:23,548 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:23,548 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:23,548 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:23,548 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:23,550 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,550 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,550 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,551 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,552 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:23,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:23,555 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,555 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,555 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,555 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,555 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:23,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,555 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:23,655 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:23,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:23,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:23,658 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:23,659 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,659 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:23,659 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:23,659 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,659 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,660 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,660 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,661 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,661 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:23,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:23,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:23,662 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:23,662 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,662 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:23,662 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:23,662 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:23,662 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,662 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:23,662 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:23,662 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,662 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:23,662 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:23,662 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,663 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:23,663 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:23,665 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,665 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,666 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,666 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:23,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22541a22[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:23,667 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:23,667 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,668 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,668 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:23,668 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:23,668 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:23,668 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,668 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:23,669 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:23,669 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,669 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,669 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:23,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:23,672 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:23,672 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:23,672 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:23,672 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:23,672 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:23,673 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,674 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,674 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,674 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,675 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,678 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,678 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,678 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,678 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,678 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,678 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:23,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:23,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:23,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,679 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:23,679 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:23,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193323686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193323690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193323707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:23,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193323715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,779 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:23,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:23,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:23,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:23,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:23,784 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:23,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:23,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:23,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,786 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,786 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,786 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,786 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:23,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:23,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:23,787 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:23,787 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,787 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:23,788 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,788 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:23,788 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:23,788 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:23,789 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,789 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,790 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,790 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:23,791 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:23,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fc279f4[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:23,791 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,795 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:23,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:23,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:23,795 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,795 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:23,797 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:23,797 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,797 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,797 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:23,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:23,800 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:23,800 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:23,800 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:23,800 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:23,800 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:23,801 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,801 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,801 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,802 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,803 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,805 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,805 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,805 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,805 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:23,805 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:23,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:23,806 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,805 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,806 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,806 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,806 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:23,806 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:23,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,875 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/3bd767519ee343a9bcfd2b819e3dbdca 2023-04-22 19:54:23,906 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:23,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:23,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:23,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:23,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:23,909 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:23,909 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,909 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:23,909 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:23,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,915 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,916 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,916 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,916 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:23,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:23,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:23,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:23,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:23,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:23,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:23,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:23,917 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,917 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:23,917 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:23,917 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:23,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:23,917 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:23,918 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,918 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:23,918 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:23,919 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,919 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,919 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,919 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:23,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ab7da74[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:23,921 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:23,921 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,922 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,922 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:23,922 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,922 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:23,922 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:23,922 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:23,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,923 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,923 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,923 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:23,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:23,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:23,925 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:23,926 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:23,926 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:23,926 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:23,926 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:23,928 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,928 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:23,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:23,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,928 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:23,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:23,929 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:23,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,930 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:23,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:23,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,932 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,932 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:23,932 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:23,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:23,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:23,932 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:23,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:23,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:23,932 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:23,932 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:23,932 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:23,932 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,932 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:23,932 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:23,932 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:23,933 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:23,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:23,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:24,033 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:24,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:24,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:24,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:24,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:24,036 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:24,036 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:24,036 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:24,036 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:24,037 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,037 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,038 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:24,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:24,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:24,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:24,039 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:24,039 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:24,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:24,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:24,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:24,040 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:24,040 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:24,040 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:24,040 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:24,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:24,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:24,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:24,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:24,042 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,042 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,042 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,042 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@204397ce[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:24,043 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:24,044 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,046 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,046 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,046 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:24,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:24,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:24,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:24,047 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:24,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:24,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:24,050 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:24,050 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:24,050 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:24,050 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:24,050 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:24,052 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,052 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,052 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,052 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,053 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,056 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:24,056 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:24,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:24,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:24,056 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,056 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,056 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,056 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,056 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,056 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,056 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:24,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:24,056 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:24,056 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:24,057 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:24,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:24,157 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:24,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:24,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:24,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:24,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:24,160 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:24,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:24,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:24,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:24,161 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,161 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,163 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:24,163 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:24,164 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,164 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:24,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:24,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:24,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:24,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:24,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:24,166 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:24,167 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:24,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22441cee[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:24,169 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:24,169 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,170 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:24,170 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:24,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:24,170 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:24,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:24,172 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:24,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,172 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:24,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:24,174 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:24,174 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:24,174 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:24,174 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:24,174 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:24,179 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,179 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,179 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,180 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,184 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:24,184 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,184 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:24,184 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:24,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:24,184 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:24,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:24,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:24,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:24,184 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:24,185 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:24,185 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:24,184 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,185 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,185 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,185 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:24,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193324190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:24,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193324196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:24,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193324210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193324221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,285 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:24,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:24,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:24,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:24,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:24,288 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:24,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:24,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:24,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:24,289 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,289 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,290 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:24,290 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,290 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,290 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:24,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,292 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/4097aeff7a6f42f0970377c9da035e76 2023-04-22 19:54:24,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:24,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:24,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:24,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:24,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:24,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:24,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:24,293 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:24,293 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:24,293 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:24,293 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:24,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/d41d54d42c854164a073f895847b7c14 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d41d54d42c854164a073f895847b7c14 2023-04-22 19:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:24,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e85dc8b[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:24,303 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:24,303 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,303 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d41d54d42c854164a073f895847b7c14, entries=400, sequenceid=184, filesize=59.4 K 2023-04-22 19:54:24,304 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/3bd767519ee343a9bcfd2b819e3dbdca as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/3bd767519ee343a9bcfd2b819e3dbdca 2023-04-22 19:54:24,306 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:24,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:24,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:24,306 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,306 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:24,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:24,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:24,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:24,310 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/3bd767519ee343a9bcfd2b819e3dbdca, entries=150, sequenceid=184, filesize=11.8 K 2023-04-22 19:54:24,311 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/4097aeff7a6f42f0970377c9da035e76 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4097aeff7a6f42f0970377c9da035e76 2023-04-22 19:54:24,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4097aeff7a6f42f0970377c9da035e76, entries=150, sequenceid=184, filesize=11.8 K 2023-04-22 19:54:24,317 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for 98809de8ceb9ac8f3bc773e820fd0e11 in 1282ms, sequenceid=184, compaction requested=true 2023-04-22 19:54:24,317 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:24,317 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:24,317 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:24,318 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:24,318 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:24,318 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:24,319 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105990 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:24,319 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/A is initiating minor compaction (all files) 2023-04-22 19:54:24,319 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/A in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:24,320 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d743495e03d9449ca5feb5145299983e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/bb1950696c92409ca86d3fdb83f8cff5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d41d54d42c854164a073f895847b7c14] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=103.5 K 2023-04-22 19:54:24,320 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting d743495e03d9449ca5feb5145299983e, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682193260549 2023-04-22 19:54:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,320 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:24,321 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/B is initiating minor compaction (all files) 2023-04-22 19:54:24,321 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/B in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:24,321 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/00a514c9d55d40579f61642f04a326f3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/b27a4fc977cd43449cdbd801bbf27044, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/3bd767519ee343a9bcfd2b819e3dbdca] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=33.3 K 2023-04-22 19:54:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,321 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting bb1950696c92409ca86d3fdb83f8cff5, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682193260867 2023-04-22 19:54:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,321 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 00a514c9d55d40579f61642f04a326f3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682193260549 2023-04-22 19:54:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,322 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting d41d54d42c854164a073f895847b7c14, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1682193263020 2023-04-22 19:54:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,323 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b27a4fc977cd43449cdbd801bbf27044, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682193260867 2023-04-22 19:54:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:24,323 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 3bd767519ee343a9bcfd2b819e3dbdca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1682193263025 2023-04-22 19:54:24,323 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:24,323 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:24,324 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:24,324 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:24,324 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:24,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:24,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:24,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,327 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,330 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,335 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,335 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,335 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:24,335 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,335 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:24,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:24,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:24,335 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,335 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,335 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,335 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,335 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:24,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:24,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:24,336 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:24,336 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:24,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:24,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:24,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:24,343 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#B#compaction#180 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:24,348 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#A#compaction#179 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:24,436 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:24,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:24,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:24,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:24,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:24,440 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:24,440 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:24,440 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:24,440 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:24,440 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,440 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,442 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:24,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:24,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:24,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:24,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:24,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:24,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:24,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:24,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:24,445 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:24,445 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:24,445 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:24,445 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:24,445 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=147.60 KB heapSize=359.20 KB 2023-04-22 19:54:24,445 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:24,445 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:24,445 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:24,445 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:24,445 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:24,446 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:24,459 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422cc4ff1bc0bb04dd6a82b601e6b38e927 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cc4ff1bc0bb04dd6a82b601e6b38e927 2023-04-22 19:54:24,766 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/0547c166feae43f0be0d82890a0f536a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/0547c166feae43f0be0d82890a0f536a 2023-04-22 19:54:24,772 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/B of 98809de8ceb9ac8f3bc773e820fd0e11 into 0547c166feae43f0be0d82890a0f536a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:24,772 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:24,772 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/B, priority=13, startTime=1682193264317; duration=0sec 2023-04-22 19:54:24,772 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:24,772 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:24,773 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:24,773 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/C is initiating minor compaction (all files) 2023-04-22 19:54:24,774 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/C in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:24,774 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/e862bb4f25f147288aa7232f884cf4b8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b27e163339434456b291b30f86a1f8cc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4097aeff7a6f42f0970377c9da035e76] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=33.3 K 2023-04-22 19:54:24,774 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting e862bb4f25f147288aa7232f884cf4b8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682193260549 2023-04-22 19:54:24,775 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b27e163339434456b291b30f86a1f8cc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682193260867 2023-04-22 19:54:24,775 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 4097aeff7a6f42f0970377c9da035e76, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1682193263025 2023-04-22 19:54:24,783 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#C#compaction#182 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:24,793 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/78b262f3830847b7951d42100311bae6 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/78b262f3830847b7951d42100311bae6 2023-04-22 19:54:24,799 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/C of 98809de8ceb9ac8f3bc773e820fd0e11 into 78b262f3830847b7951d42100311bae6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:24,799 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:24,799 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/C, priority=13, startTime=1682193264318; duration=0sec 2023-04-22 19:54:24,799 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:24,865 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=210, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/d2d8939d76a043d1b6b2b9c1d7fce671 2023-04-22 19:54:24,894 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/8ee4bff1edb14a54ad866decff72fc7b 2023-04-22 19:54:24,945 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/f8aca192fb06400bb3bc6f27e17297f3 2023-04-22 19:54:24,951 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/d2d8939d76a043d1b6b2b9c1d7fce671 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d2d8939d76a043d1b6b2b9c1d7fce671 2023-04-22 19:54:24,958 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d2d8939d76a043d1b6b2b9c1d7fce671, entries=150, sequenceid=210, filesize=25.3 K 2023-04-22 19:54:24,959 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/8ee4bff1edb14a54ad866decff72fc7b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/8ee4bff1edb14a54ad866decff72fc7b 2023-04-22 19:54:24,967 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/8ee4bff1edb14a54ad866decff72fc7b, entries=150, sequenceid=210, filesize=11.8 K 2023-04-22 19:54:24,968 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/f8aca192fb06400bb3bc6f27e17297f3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f8aca192fb06400bb3bc6f27e17297f3 2023-04-22 19:54:24,979 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f8aca192fb06400bb3bc6f27e17297f3, entries=150, sequenceid=210, filesize=11.8 K 2023-04-22 19:54:24,980 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~359.16 KB/367776, currentSize=0 B/0 for 98809de8ceb9ac8f3bc773e820fd0e11 in 535ms, sequenceid=210, compaction requested=false 2023-04-22 19:54:24,980 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:24,980 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:24,982 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,982 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7af5cf55[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:24,985 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:24,985 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,986 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:24,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:24,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:24,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:24,991 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:24,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:24,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:24,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:24,991 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:24,994 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,994 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:24,994 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:24,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:24,995 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:24,996 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:24,999 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,999 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,999 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:24,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:24,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:24,999 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:24,999 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:24,999 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:24,999 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,000 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,000 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:24,999 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,000 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:25,000 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:24,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:25,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,100 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:25,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:25,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:25,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:25,104 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:25,104 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,104 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:25,104 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:25,105 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,105 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,107 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,107 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:25,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:25,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:25,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:25,109 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,109 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:25,109 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:25,109 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:25,111 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4634da22[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:25,114 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:25,114 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,118 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:25,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,119 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:25,120 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:25,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:25,124 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:25,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:25,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:25,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:25,124 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:25,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,127 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,128 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,134 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,134 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,134 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:25,134 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,134 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,134 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:25,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:25,134 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,135 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,135 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,135 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,135 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:25,135 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:25,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,187 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/13bf49d6ccff4a999cb7ddf01408ef3e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/13bf49d6ccff4a999cb7ddf01408ef3e 2023-04-22 19:54:25,193 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/A of 98809de8ceb9ac8f3bc773e820fd0e11 into 13bf49d6ccff4a999cb7ddf01408ef3e(size=25.7 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:25,193 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:25,193 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/A, priority=13, startTime=1682193264317; duration=0sec 2023-04-22 19:54:25,193 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:25,235 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:25,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:25,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:25,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:25,238 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:54:25,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:25,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:25,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:25,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:25,239 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:25,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:25,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:25,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:25,240 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:25,240 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,240 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:25,240 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:25,241 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,241 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,242 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,243 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,243 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:25,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:25,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:25,268 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,268 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:25,268 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:25,268 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,268 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:25,268 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:25,268 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,272 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:25,272 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,274 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44151d66[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:25,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:25,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,278 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,278 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,278 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:25,279 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:25,279 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:25,281 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:25,281 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,281 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:25,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:25,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:25,285 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:25,285 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:25,285 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:25,285 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:25,285 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:25,287 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,287 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,287 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,287 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:25,291 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,291 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:25,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,291 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:25,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:25,293 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422948388b1ac484a41abb5bd8f98669101 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422948388b1ac484a41abb5bd8f98669101 2023-04-22 19:54:25,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193325280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193265276, max timestamp=1682193265285Num uniques 859; 2023-04-22 19:54:25,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193265276, max timestamp=1682193265285Num uniques 859; 2023-04-22 19:54:25,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:25,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:25,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193265276, max timestamp=1682193265283Num uniques 859; 2023-04-22 19:54:25,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:25,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193325344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43838 deadline: 1682193325347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193325348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,392 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:25,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:25,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:25,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:25,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:25,395 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:25,395 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,395 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:25,395 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:25,396 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,396 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,397 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,397 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,397 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:25,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:25,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:25,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:25,398 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:25,398 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,398 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:25,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:25,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:25,399 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,399 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:25,399 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:25,399 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:25,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:25,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:25,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:25,402 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,402 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,402 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,402 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:25,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@434e0eb[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:25,404 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:25,404 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,405 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:25,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:25,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:25,406 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,406 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:25,407 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,407 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,407 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,407 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:25,409 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:25,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:25,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:25,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:25,410 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:25,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193325409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,412 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,413 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,416 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:25,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:25,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:25,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,417 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:25,417 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:25,417 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43838 deadline: 1682193325449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43802 deadline: 1682193325449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43826 deadline: 1682193325449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,517 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:25,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:25,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:25,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:25,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:25,520 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:25,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:25,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:25,521 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,521 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,522 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,522 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,522 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,522 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:25,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:25,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:25,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:25,523 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:25,523 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:25,524 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,524 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:25,524 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:25,524 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:25,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,526 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,527 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,527 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:25,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@506f185f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:25,529 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:25,529 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,530 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,530 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:25,530 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:25,530 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:25,530 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,530 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:25,532 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:25,532 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,532 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,532 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:25,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:25,535 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:25,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:25,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:25,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:25,535 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:25,536 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,536 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,536 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,537 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,539 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,543 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,543 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,543 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,543 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,543 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,543 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:25,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:25,543 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,543 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,543 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,543 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:25,544 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:25,544 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:25,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193325613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,644 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:25,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:25,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:25,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:25,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:25,647 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:25,647 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,647 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:25,648 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:25,648 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,648 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,650 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,650 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:25,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:25,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:25,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:25,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:25,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,653 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43802 deadline: 1682193325652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,654 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:25,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43838 deadline: 1682193325651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:25,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,654 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:25,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43826 deadline: 1682193325653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,654 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:25,654 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,654 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:25,654 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:25,654 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,654 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:25,654 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:25,655 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,655 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,656 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,656 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:25,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@306a3dba[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:25,658 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:25,658 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,659 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,659 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:25,659 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:25,659 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:25,659 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,659 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:25,660 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:25,660 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,660 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,661 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:25,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:25,664 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:25,664 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:25,664 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:25,664 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:25,664 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:25,666 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,666 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,666 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,667 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,668 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,670 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,670 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,670 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,670 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:25,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:25,671 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,671 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,671 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,671 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:25,671 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:25,671 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,671 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,671 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,698 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=224, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/04a9185fbc734aa48fc38e0f5a823d6a 2023-04-22 19:54:25,771 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:25,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:25,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:25,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:25,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:25,774 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:25,774 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:25,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:25,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,776 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,776 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,777 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,777 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:25,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:25,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:25,778 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,778 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:25,779 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:25,779 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:25,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,780 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,781 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,781 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:25,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf8f8fc[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:25,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:25,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,785 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:25,785 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:25,785 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:25,785 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,785 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:25,786 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,786 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:25,786 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,786 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:25,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:25,793 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:25,793 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:25,793 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:25,793 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:25,793 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:25,794 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,794 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,794 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,796 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:25,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:25,808 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,809 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,808 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:25,809 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:25,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,909 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:25,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:25,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:25,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:25,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:25,912 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:25,912 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,912 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:25,912 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:25,912 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,913 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,914 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:25,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:25,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:25,915 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:25,915 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:25,915 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:25,915 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:25,915 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:25,915 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,915 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:25,916 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:25,916 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:25,916 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:25,916 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:25,916 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,916 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:25,916 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:25,918 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,918 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,919 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,919 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:25,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fe86830[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:25,920 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:25,920 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193325918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,921 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,921 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:25,921 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:25,921 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:25,921 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,921 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:25,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,923 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,923 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,923 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:25,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:25,925 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:25,925 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:25,925 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:25,926 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:25,926 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:25,927 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,927 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:25,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,927 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:25,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,928 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,929 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,933 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,933 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:25,933 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:25,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:25,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:25,933 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:25,933 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:25,933 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:25,934 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:25,934 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,934 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:25,934 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,934 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:25,934 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:25,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:25,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43802 deadline: 1682193325957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43838 deadline: 1682193325957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:25,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:25,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43826 deadline: 1682193325957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,034 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:26,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:26,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:26,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:26,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:26,038 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:26,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:26,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:26,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:26,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,039 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:26,039 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:26,040 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,040 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:26,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:26,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:26,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:26,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:26,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:26,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:26,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:26,041 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,041 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:26,041 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:26,041 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:26,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:26,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:26,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:26,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,045 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,045 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:26,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@576d3a72[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:26,047 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:26,047 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,052 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:26,052 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,052 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:26,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:26,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:26,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:26,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:26,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:26,056 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:26,056 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:26,056 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:26,056 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:26,056 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:26,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:26,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,065 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:26,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:26,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,068 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:26,069 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,069 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:26,069 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:26,069 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:26,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/818bc3c68fe445f99eb64a1aa4e7e4b2 2023-04-22 19:54:26,169 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:26,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:26,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:26,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:26,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:26,175 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:26,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:26,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:26,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:26,176 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,176 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,178 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:26,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,180 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:26,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:26,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:26,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:26,181 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,181 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:26,181 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:26,181 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:26,183 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,183 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:26,185 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:26,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@360b4ac8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:26,185 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,186 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,186 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:26,186 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:26,186 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,186 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:26,186 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:26,188 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,188 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:26,188 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,188 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:26,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:26,190 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:26,190 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:26,190 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:26,191 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:26,191 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:26,193 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:26,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:26,194 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,195 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:26,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:26,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:26,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,197 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:26,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:26,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:26,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,198 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,198 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:26,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:26,198 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:26,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:26,298 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:26,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:26,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:26,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:26,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:26,302 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:26,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:26,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:26,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:26,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:26,303 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:26,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:26,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:26,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:26,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:26,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:26,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:26,305 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,306 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:26,306 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:26,306 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:26,307 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,307 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:26,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d2f6bed[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:26,310 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:26,310 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,312 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,313 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:26,313 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:26,313 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:26,313 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,313 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:26,314 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:26,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:26,317 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:26,317 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:26,317 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:26,317 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:26,317 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:26,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:26,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,322 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,324 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,324 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:26,324 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:26,325 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:26,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:26,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:26,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:26,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,325 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:26,325 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:26,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:26,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:26,425 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:26,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:26,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:26,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:26,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:26,428 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:26,428 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:26,428 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:26,428 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:26,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:26,429 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43766 deadline: 1682193326426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,429 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,431 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:26,431 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:26,432 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,432 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:26,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:26,433 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,433 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:26,433 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:26,433 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:26,435 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,435 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,435 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,435 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:26,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@113a2f90[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:26,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:26,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,440 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:26,440 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,440 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:26,443 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:26,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:26,446 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:26,446 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:26,446 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:26,446 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:26,446 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:26,447 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,447 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:26,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,447 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:26,448 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,452 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,452 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,452 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,452 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,452 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,452 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:26,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:26,452 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:26,452 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:26,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:26,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:26,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:26,452 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:26,452 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:26,453 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:26,453 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:26,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:26,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:26,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43826 deadline: 1682193326460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:26,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43838 deadline: 1682193326462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:26,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43802 deadline: 1682193326462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/2de212aefcc7411abb18bb6d4ba24e38 2023-04-22 19:54:26,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/04a9185fbc734aa48fc38e0f5a823d6a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/04a9185fbc734aa48fc38e0f5a823d6a 2023-04-22 19:54:26,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/04a9185fbc734aa48fc38e0f5a823d6a, entries=350, sequenceid=224, filesize=52.6 K 2023-04-22 19:54:26,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/818bc3c68fe445f99eb64a1aa4e7e4b2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/818bc3c68fe445f99eb64a1aa4e7e4b2 2023-04-22 19:54:26,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/818bc3c68fe445f99eb64a1aa4e7e4b2, entries=100, sequenceid=224, filesize=9.4 K 2023-04-22 19:54:26,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/2de212aefcc7411abb18bb6d4ba24e38 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/2de212aefcc7411abb18bb6d4ba24e38 2023-04-22 19:54:26,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/2de212aefcc7411abb18bb6d4ba24e38, entries=100, sequenceid=224, filesize=9.4 K 2023-04-22 19:54:26,539 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 98809de8ceb9ac8f3bc773e820fd0e11 in 1301ms, sequenceid=224, compaction requested=true 2023-04-22 19:54:26,540 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:26,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:26,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:26,540 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:26,540 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:26,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:26,541 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34158 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:26,541 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106092 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:26,541 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/B is initiating minor compaction (all files) 2023-04-22 19:54:26,541 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/A is initiating minor compaction (all files) 2023-04-22 19:54:26,541 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/B in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,541 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/A in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,541 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/0547c166feae43f0be0d82890a0f536a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/8ee4bff1edb14a54ad866decff72fc7b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/818bc3c68fe445f99eb64a1aa4e7e4b2] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=33.4 K 2023-04-22 19:54:26,541 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/13bf49d6ccff4a999cb7ddf01408ef3e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d2d8939d76a043d1b6b2b9c1d7fce671, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/04a9185fbc734aa48fc38e0f5a823d6a] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=103.6 K 2023-04-22 19:54:26,542 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0547c166feae43f0be0d82890a0f536a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1682193263025 2023-04-22 19:54:26,542 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 13bf49d6ccff4a999cb7ddf01408ef3e, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1682193263025 2023-04-22 19:54:26,542 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting d2d8939d76a043d1b6b2b9c1d7fce671, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682193263088 2023-04-22 19:54:26,542 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 8ee4bff1edb14a54ad866decff72fc7b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682193263088 2023-04-22 19:54:26,543 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 04a9185fbc734aa48fc38e0f5a823d6a, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682193265208 2023-04-22 19:54:26,543 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 818bc3c68fe445f99eb64a1aa4e7e4b2, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682193265231 2023-04-22 19:54:26,552 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#B#compaction#188 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:26,553 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:26,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:26,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:26,566 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#A#compaction#189 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:26,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:26,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:26,567 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:26,567 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:26,567 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:26,567 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:26,567 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,567 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:26,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:26,569 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,570 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:26,570 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:26,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:26,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:26,574 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/9871dbb72af54ffc94e745ab88697172 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/9871dbb72af54ffc94e745ab88697172 2023-04-22 19:54:26,581 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/B of 98809de8ceb9ac8f3bc773e820fd0e11 into 9871dbb72af54ffc94e745ab88697172(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:26,581 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:26,581 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/B, priority=13, startTime=1682193266540; duration=0sec 2023-04-22 19:54:26,581 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:26,581 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:26,582 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34158 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:26,583 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/C is initiating minor compaction (all files) 2023-04-22 19:54:26,583 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/C in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,583 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/78b262f3830847b7951d42100311bae6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f8aca192fb06400bb3bc6f27e17297f3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/2de212aefcc7411abb18bb6d4ba24e38] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=33.4 K 2023-04-22 19:54:26,583 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 78b262f3830847b7951d42100311bae6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1682193263025 2023-04-22 19:54:26,584 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f8aca192fb06400bb3bc6f27e17297f3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682193263088 2023-04-22 19:54:26,584 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 2de212aefcc7411abb18bb6d4ba24e38, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682193265231 2023-04-22 19:54:26,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:26,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:26,588 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:26,588 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:26,588 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:26,592 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:26,592 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:26,592 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:26,592 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:26,592 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-22 19:54:26,592 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:26,593 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:26,593 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:26,593 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:26,593 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:26,593 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:26,597 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#C#compaction#190 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:27,014 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/5068b45ebfa540fba081ebbcbd432370 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/5068b45ebfa540fba081ebbcbd432370 2023-04-22 19:54:27,020 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/C of 98809de8ceb9ac8f3bc773e820fd0e11 into 5068b45ebfa540fba081ebbcbd432370(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:27,020 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:27,020 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/C, priority=13, startTime=1682193266540; duration=0sec 2023-04-22 19:54:27,020 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:27,024 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422f86149da925d4adb9fffc72330326da6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f86149da925d4adb9fffc72330326da6 2023-04-22 19:54:27,033 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=262, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/65bc9f1f21544e5abd6bae4e950244d6 2023-04-22 19:54:27,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:27,438 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:27,444 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/5c9aef63ec22406bb27b90078175943d 2023-04-22 19:54:27,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:27,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193327453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:27,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43826 deadline: 1682193327463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:27,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43802 deadline: 1682193327465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:27,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43838 deadline: 1682193327473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:27,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193327556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:27,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193327761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,806 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/3ae36414b8d247768c0b8acdf5a1e49d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/3ae36414b8d247768c0b8acdf5a1e49d 2023-04-22 19:54:27,812 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/A of 98809de8ceb9ac8f3bc773e820fd0e11 into 3ae36414b8d247768c0b8acdf5a1e49d(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-22 19:54:27,812 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:27,812 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/A, priority=13, startTime=1682193266540; duration=1sec 2023-04-22 19:54:27,812 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:27,855 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/59c0367f75444e8188d89f4b92d282f7 2023-04-22 19:54:27,860 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/65bc9f1f21544e5abd6bae4e950244d6 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/65bc9f1f21544e5abd6bae4e950244d6 2023-04-22 19:54:27,864 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/65bc9f1f21544e5abd6bae4e950244d6, entries=150, sequenceid=262, filesize=25.4 K 2023-04-22 19:54:27,865 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/5c9aef63ec22406bb27b90078175943d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/5c9aef63ec22406bb27b90078175943d 2023-04-22 19:54:27,869 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/5c9aef63ec22406bb27b90078175943d, entries=150, sequenceid=262, filesize=11.9 K 2023-04-22 19:54:27,870 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/59c0367f75444e8188d89f4b92d282f7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/59c0367f75444e8188d89f4b92d282f7 2023-04-22 19:54:27,877 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/59c0367f75444e8188d89f4b92d282f7, entries=150, sequenceid=262, filesize=11.9 K 2023-04-22 19:54:27,878 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 98809de8ceb9ac8f3bc773e820fd0e11 in 1286ms, sequenceid=262, compaction requested=false 2023-04-22 19:54:27,878 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:27,879 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:27,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,882 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:27,882 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:27,882 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:27,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:27,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3a296f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:27,884 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:27,884 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:27,885 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:27,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:27,885 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:27,885 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:27,885 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:27,885 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,885 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:27,886 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,887 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:27,887 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:27,887 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:27,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:27,889 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:27,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:27,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:27,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:27,889 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:27,891 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:27,891 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:27,891 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:27,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:27,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:27,892 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:27,893 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,900 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,900 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:27,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:27,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:27,900 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:27,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:27,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:27,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:27,900 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:27,901 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:27,901 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:27,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:28,001 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:28,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:28,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:28,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:28,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:28,004 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:28,005 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:28,005 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:28,005 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:28,005 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:28,005 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:28,006 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:28,006 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:28,007 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,007 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:28,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:28,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:28,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:28,008 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:28,008 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:28,008 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:54:28,009 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:28,009 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:28,009 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:28,009 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:28,009 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:28,009 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:28,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:28,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:28,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193268093, max timestamp=1682193268124Num uniques 859; 2023-04-22 19:54:28,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:54:28,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193268093, max timestamp=1682193268124Num uniques 859; 2023-04-22 19:54:28,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:54:28,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193268093, max timestamp=1682193268124Num uniques 859; 2023-04-22 19:54:28,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:54:28,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:28,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193328169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:28,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193328273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,423 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304223275e56c7e454d0c800a2cd4bf9093dc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223275e56c7e454d0c800a2cd4bf9093dc 2023-04-22 19:54:28,431 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=274, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/4d08d72274764d72b5bb39292e94efe4 2023-04-22 19:54:28,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:28,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193328477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:28,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193328781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,843 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/ed018bba9dc6452085c8ce50878bfdd5 2023-04-22 19:54:28,854 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/efc8e8fcc94b452b9f407c66867b02e5 2023-04-22 19:54:28,858 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/4d08d72274764d72b5bb39292e94efe4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/4d08d72274764d72b5bb39292e94efe4 2023-04-22 19:54:28,862 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/4d08d72274764d72b5bb39292e94efe4, entries=150, sequenceid=274, filesize=25.4 K 2023-04-22 19:54:28,863 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/ed018bba9dc6452085c8ce50878bfdd5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/ed018bba9dc6452085c8ce50878bfdd5 2023-04-22 19:54:28,867 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/ed018bba9dc6452085c8ce50878bfdd5, entries=150, sequenceid=274, filesize=11.9 K 2023-04-22 19:54:28,868 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/efc8e8fcc94b452b9f407c66867b02e5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/efc8e8fcc94b452b9f407c66867b02e5 2023-04-22 19:54:28,871 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/efc8e8fcc94b452b9f407c66867b02e5, entries=150, sequenceid=274, filesize=11.9 K 2023-04-22 19:54:28,872 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 98809de8ceb9ac8f3bc773e820fd0e11 in 864ms, sequenceid=274, compaction requested=true 2023-04-22 19:54:28,872 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:28,872 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:28,874 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,874 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:28,874 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:28,874 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:28,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:28,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@430b7b4a[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:28,876 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:28,876 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:28,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:28,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:28,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:28,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:28,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:28,877 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,877 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:28,879 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:28,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:28,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:28,882 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:28,882 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:28,882 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:28,882 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:28,882 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:28,883 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:28,883 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:28,883 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:28,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:28,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:28,885 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,886 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,888 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:28,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:28,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:28,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:28,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:28,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:28,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:28,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:28,889 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:28,889 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:28,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:28,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:28,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:28,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:28,989 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:28,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:28,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:28,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:28,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:28,992 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:28,993 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:28,993 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:28,993 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:28,993 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:28,993 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:28,994 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:28,994 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:28,995 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:28,995 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:28,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:28,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:28,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:28,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:28,996 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:28,996 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:28,996 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:28,996 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:28,996 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:28,996 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:28,996 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:28,997 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-22 19:54:28,997 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:28,997 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:28,997 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:28,997 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:28,997 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:28,997 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:29,017 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042225e60ce9e2d04f4cb2ffd4ba8cd79b28 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042225e60ce9e2d04f4cb2ffd4ba8cd79b28 2023-04-22 19:54:29,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:29,288 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:29,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193329304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193329406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,423 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=309, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/326ad17e3b7e450092fbd27da6ff5bd2 2023-04-22 19:54:29,433 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/56961b29c6de4ddf8c06fa024f9480ad 2023-04-22 19:54:29,444 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/123f31625103455d9c78c74d88237fe3 2023-04-22 19:54:29,450 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/326ad17e3b7e450092fbd27da6ff5bd2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/326ad17e3b7e450092fbd27da6ff5bd2 2023-04-22 19:54:29,454 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/326ad17e3b7e450092fbd27da6ff5bd2, entries=150, sequenceid=309, filesize=25.4 K 2023-04-22 19:54:29,455 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/56961b29c6de4ddf8c06fa024f9480ad as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/56961b29c6de4ddf8c06fa024f9480ad 2023-04-22 19:54:29,459 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/56961b29c6de4ddf8c06fa024f9480ad, entries=150, sequenceid=309, filesize=11.9 K 2023-04-22 19:54:29,460 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/123f31625103455d9c78c74d88237fe3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/123f31625103455d9c78c74d88237fe3 2023-04-22 19:54:29,464 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/123f31625103455d9c78c74d88237fe3, entries=150, sequenceid=309, filesize=11.9 K 2023-04-22 19:54:29,464 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 98809de8ceb9ac8f3bc773e820fd0e11 in 468ms, sequenceid=309, compaction requested=true 2023-04-22 19:54:29,465 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:29,465 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:29,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:29,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:29,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:29,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:29,467 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,467 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:29,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:29,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:29,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 19:54:29,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152d00c5[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:29,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:29,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:29,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:29,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:29,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:29,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:29,471 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:29,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:29,477 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:29,476 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:29,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:29,480 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:29,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:29,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:29,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:29,480 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:29,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,490 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,493 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422c0543b14238a4a239a5becfb050073f6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c0543b14238a4a239a5becfb050073f6 2023-04-22 19:54:29,493 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:29,494 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:29,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:29,494 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,494 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,494 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,494 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,494 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,494 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,494 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:29,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:29,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:29,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:29,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:29,495 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:29,495 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:29,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682193269487, max timestamp=1682193269505Num uniques 859; 2023-04-22 19:54:29,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682193269487, max timestamp=1682193269505Num uniques 859; 2023-04-22 19:54:29,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-22 19:54:29,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-22 19:54:29,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682193269487, max timestamp=1682193269505Num uniques 859; 2023-04-22 19:54:29,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-22 19:54:29,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43826 deadline: 1682193329523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43838 deadline: 1682193329525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43802 deadline: 1682193329528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,595 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:29,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:29,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:29,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:29,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:29,598 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:29,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:29,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:29,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:29,599 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,599 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,600 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:29,600 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:29,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:29,600 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,600 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:29,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:29,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:29,602 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:29,602 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:29,602 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:29,602 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:29,604 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,604 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,604 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,604 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@552beb5f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:29,606 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:29,606 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,608 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,608 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,608 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:29,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:29,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:29,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:29,609 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,609 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,609 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:29,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:29,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:29,612 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:29,612 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:29,612 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:29,612 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:29,612 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:29,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,613 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193329610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,613 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:29,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:29,614 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,618 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,621 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,621 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:29,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:29,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:29,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:29,621 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:29,621 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:29,621 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,621 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:29,621 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:29,621 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,621 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,621 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,621 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:29,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43826 deadline: 1682193329629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43838 deadline: 1682193329631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43802 deadline: 1682193329635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,721 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:29,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:29,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:29,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:29,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:29,724 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:29,724 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:29,724 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:29,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:29,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,728 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,728 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:29,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:29,728 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,728 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:29,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:29,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:29,729 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:29,729 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:29,729 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:29,729 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:29,731 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,731 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,731 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,731 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:29,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba4c1c4[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:29,732 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:29,733 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,734 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,734 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:29,734 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,734 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:29,734 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:29,734 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:29,735 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:29,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:29,737 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:29,737 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:29,737 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:29,737 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:29,738 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:29,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,741 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,742 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,744 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:29,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:29,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:29,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:29,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:29,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:29,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:29,745 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:29,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:29,745 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:29,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43826 deadline: 1682193329830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43838 deadline: 1682193329836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43802 deadline: 1682193329837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,845 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:29,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:29,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:29,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:29,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:29,848 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:29,848 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:29,848 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:29,848 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:29,848 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,849 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,850 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:29,850 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:29,850 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,850 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:29,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:29,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:29,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:29,851 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:29,851 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:29,851 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:29,851 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:29,851 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:29,851 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:29,851 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:29,852 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:29,852 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:29,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:29,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:29,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:29,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:29,853 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:29,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@336d2ed1[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:29,855 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:29,855 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,857 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,857 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,857 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:29,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,858 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:29,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,858 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:29,858 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:29,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:29,861 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:29,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:29,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:29,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:29,861 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:29,862 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,862 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,862 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:29,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:29,866 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,869 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,879 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:29,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:29,879 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:29,879 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:29,879 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:29,879 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,879 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,879 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,879 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:29,879 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:29,879 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:29,879 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,879 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:29,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:29,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:29,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193329914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,919 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=320, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/c19f73ff282f4d39b72e59d5829a44c3 2023-04-22 19:54:29,979 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:29,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:29,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:29,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:29,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:29,983 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:29,983 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:29,983 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:29,983 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:29,984 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,984 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,985 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:29,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:29,985 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:29,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:29,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:29,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:29,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:29,987 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:29,987 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:29,987 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:29,987 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:29,989 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,989 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,989 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,989 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:29,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e3b3b51[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:29,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:29,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,992 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:29,993 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,993 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:29,995 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:29,995 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,995 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:29,995 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:29,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:29,998 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:29,998 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:29,998 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:29,998 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:29,998 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:30,000 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,000 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,004 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:30,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:30,007 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,007 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:30,007 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:30,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,007 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:30,008 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,008 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,107 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:30,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:30,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:30,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:30,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:30,116 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:30,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:30,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:30,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,118 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:30,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:30,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:30,120 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,120 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:30,120 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:30,120 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:30,124 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,124 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,126 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@444040da[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:30,131 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:30,131 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,132 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,133 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:30,133 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:30,133 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:30,133 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,133 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:30,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:30,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43826 deadline: 1682193330134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,135 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:30,135 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,135 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,135 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:30,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:30,138 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:30,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:30,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:30,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:30,138 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:30,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,144 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,144 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,144 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,144 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:30,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:30,144 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,144 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,144 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,145 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,145 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,145 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:30,145 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:30,145 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:30,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43838 deadline: 1682193330143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:30,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43802 deadline: 1682193330145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,245 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:30,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:30,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:30,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:30,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:30,249 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:30,249 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,249 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:30,249 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:30,249 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,249 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,251 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,251 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,251 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,251 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:30,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:30,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:30,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:30,252 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:30,252 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,252 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:30,253 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,253 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:30,253 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:30,253 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:30,254 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,254 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,255 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,255 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:30,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49671377[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:30,256 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:30,256 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,258 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,258 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:30,258 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:30,258 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:30,258 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,258 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:30,260 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:30,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:30,263 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:30,263 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:30,263 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:30,263 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:30,263 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:30,264 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,264 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,265 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,265 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,267 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,271 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,272 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,272 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,272 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:30,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:30,272 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,272 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:30,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,272 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:30,272 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,272 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,272 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,272 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,272 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/dc7b2638579e4bc2a63e0005f6c84aee 2023-04-22 19:54:30,372 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:30,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:30,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:30,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:30,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:30,376 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:30,376 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,376 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:30,376 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:30,376 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,376 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,379 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,379 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,379 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,379 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:30,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:30,382 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,382 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:30,382 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:30,382 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,384 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13a6a72c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:30,387 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:30,387 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,388 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:30,389 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,389 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:30,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:30,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,390 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:30,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:30,392 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:30,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:30,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:30,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:30,393 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:30,395 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,395 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,395 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,396 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,397 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,399 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,399 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,399 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,399 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,399 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,399 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,399 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,399 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:30,399 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,399 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:30,399 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,399 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:30,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:30,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193330420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,500 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:30,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:30,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:30,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:30,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:30,503 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:30,503 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,503 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:30,503 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:30,504 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,504 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,505 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,505 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,506 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,506 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:30,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:30,507 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,507 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:30,507 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:30,507 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,509 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:30,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@450e7ba[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:30,511 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:30,511 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,513 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:30,513 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,513 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:30,514 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:30,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:30,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:30,517 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:30,517 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:30,517 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:30,517 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:30,517 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:30,518 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,518 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,518 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,519 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,522 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,522 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,522 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,522 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,522 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,522 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,522 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,523 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,523 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:30,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,523 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:30,523 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:30,523 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:30,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,623 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:30,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:30,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:30,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:30,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:30,627 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:30,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:30,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:30,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,629 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,629 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,630 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,630 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:30,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:30,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:30,634 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,634 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:30,634 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:30,634 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:30,636 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:30,638 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42d0217a[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:30,638 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,639 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,639 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:30,639 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,639 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:30,639 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:30,639 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:30,640 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,640 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:30,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,640 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,641 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:30,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43826 deadline: 1682193330638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:30,643 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:30,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:30,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:30,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:30,643 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:30,647 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,647 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,647 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,647 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,653 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:30,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:30,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,654 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:30,654 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:30,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:30,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43838 deadline: 1682193330650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:30,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43802 deadline: 1682193330650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/1c2c3f1283f542cfa32b4dab4329ccd2 2023-04-22 19:54:30,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/c19f73ff282f4d39b72e59d5829a44c3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c19f73ff282f4d39b72e59d5829a44c3 2023-04-22 19:54:30,754 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:30,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:30,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c19f73ff282f4d39b72e59d5829a44c3, entries=150, sequenceid=320, filesize=25.4 K 2023-04-22 19:54:30,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/dc7b2638579e4bc2a63e0005f6c84aee as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dc7b2638579e4bc2a63e0005f6c84aee 2023-04-22 19:54:30,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:30,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:30,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:30,761 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:30,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:30,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:30,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dc7b2638579e4bc2a63e0005f6c84aee, entries=150, sequenceid=320, filesize=11.9 K 2023-04-22 19:54:30,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/1c2c3f1283f542cfa32b4dab4329ccd2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/1c2c3f1283f542cfa32b4dab4329ccd2 2023-04-22 19:54:30,765 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,765 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,766 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,766 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:30,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:30,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:30,767 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:30,767 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,767 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:30,768 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,768 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:30,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/1c2c3f1283f542cfa32b4dab4329ccd2, entries=150, sequenceid=320, filesize=11.9 K 2023-04-22 19:54:30,768 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:30,769 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:30,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:30,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:30,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:30,769 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 98809de8ceb9ac8f3bc773e820fd0e11 in 1300ms, sequenceid=320, compaction requested=true 2023-04-22 19:54:30,769 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:30,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:30,769 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:54:30,770 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:30,770 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:54:30,770 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:30,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,773 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130576 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:54:30,773 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/A is initiating minor compaction (all files) 2023-04-22 19:54:30,773 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/A in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,773 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/3ae36414b8d247768c0b8acdf5a1e49d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/65bc9f1f21544e5abd6bae4e950244d6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/4d08d72274764d72b5bb39292e94efe4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/326ad17e3b7e450092fbd27da6ff5bd2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c19f73ff282f4d39b72e59d5829a44c3] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=127.5 K 2023-04-22 19:54:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,773 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 3ae36414b8d247768c0b8acdf5a1e49d, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682193263088 2023-04-22 19:54:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,774 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61276 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:54:30,774 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/B is initiating minor compaction (all files) 2023-04-22 19:54:30,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:30,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@335d7efc[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:30,775 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/B in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:30,777 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,777 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/9871dbb72af54ffc94e745ab88697172, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/5c9aef63ec22406bb27b90078175943d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/ed018bba9dc6452085c8ce50878bfdd5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/56961b29c6de4ddf8c06fa024f9480ad, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dc7b2638579e4bc2a63e0005f6c84aee] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=59.8 K 2023-04-22 19:54:30,777 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 65bc9f1f21544e5abd6bae4e950244d6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682193265342 2023-04-22 19:54:30,777 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 9871dbb72af54ffc94e745ab88697172, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682193263088 2023-04-22 19:54:30,778 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 4d08d72274764d72b5bb39292e94efe4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1682193267446 2023-04-22 19:54:30,778 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 5c9aef63ec22406bb27b90078175943d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682193265342 2023-04-22 19:54:30,778 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:30,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:30,778 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ed018bba9dc6452085c8ce50878bfdd5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1682193267446 2023-04-22 19:54:30,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:30,778 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,779 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:30,779 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 326ad17e3b7e450092fbd27da6ff5bd2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682193268163 2023-04-22 19:54:30,779 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 56961b29c6de4ddf8c06fa024f9480ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682193268163 2023-04-22 19:54:30,779 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting dc7b2638579e4bc2a63e0005f6c84aee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1682193269287 2023-04-22 19:54:30,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:30,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:30,780 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,782 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c19f73ff282f4d39b72e59d5829a44c3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1682193269287 2023-04-22 19:54:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:30,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:30,784 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:30,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:30,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:30,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:30,784 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:30,787 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:30,787 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,787 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:30,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,788 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,790 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,795 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,795 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:30,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,795 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,796 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,796 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,796 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:30,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:30,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:30,796 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,796 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,796 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:30,796 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:30,796 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:30,796 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:30,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:30,802 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#B#compaction#203 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:30,834 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#A#compaction#204 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:30,870 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/8a9fa690b39c4d008c1aeea496711aef as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/8a9fa690b39c4d008c1aeea496711aef 2023-04-22 19:54:30,877 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/B of 98809de8ceb9ac8f3bc773e820fd0e11 into 8a9fa690b39c4d008c1aeea496711aef(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:30,877 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:30,877 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/B, priority=11, startTime=1682193270769; duration=0sec 2023-04-22 19:54:30,878 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:30,878 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:54:30,879 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61276 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:54:30,879 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 98809de8ceb9ac8f3bc773e820fd0e11/C is initiating minor compaction (all files) 2023-04-22 19:54:30,879 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98809de8ceb9ac8f3bc773e820fd0e11/C in TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,880 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/5068b45ebfa540fba081ebbcbd432370, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/59c0367f75444e8188d89f4b92d282f7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/efc8e8fcc94b452b9f407c66867b02e5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/123f31625103455d9c78c74d88237fe3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/1c2c3f1283f542cfa32b4dab4329ccd2] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp, totalSize=59.8 K 2023-04-22 19:54:30,880 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 5068b45ebfa540fba081ebbcbd432370, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682193263088 2023-04-22 19:54:30,880 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 59c0367f75444e8188d89f4b92d282f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682193265342 2023-04-22 19:54:30,881 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting efc8e8fcc94b452b9f407c66867b02e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1682193267446 2023-04-22 19:54:30,881 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 123f31625103455d9c78c74d88237fe3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682193268163 2023-04-22 19:54:30,881 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1c2c3f1283f542cfa32b4dab4329ccd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1682193269287 2023-04-22 19:54:30,896 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:30,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:30,902 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 98809de8ceb9ac8f3bc773e820fd0e11#C#compaction#205 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:30,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:30,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:30,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:30,916 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:30,916 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,916 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:30,916 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:30,918 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,918 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,919 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:30,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:30,919 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,919 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:30,920 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:30,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:30,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:30,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:30,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:30,936 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:30,936 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:30,936 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:30,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:30,950 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:30,950 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:30,950 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:30,950 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-22 19:54:30,950 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:30,950 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:30,950 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:30,950 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:30,950 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:30,950 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:30,967 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/c00ed7a63d424758b55f1745a060a8bd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c00ed7a63d424758b55f1745a060a8bd 2023-04-22 19:54:30,974 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/A of 98809de8ceb9ac8f3bc773e820fd0e11 into c00ed7a63d424758b55f1745a060a8bd(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:30,974 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:30,974 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/A, priority=11, startTime=1682193270769; duration=0sec 2023-04-22 19:54:30,974 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:30,984 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/5c0adfe2e64848d9859750bb8de2ff06 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/5c0adfe2e64848d9859750bb8de2ff06 2023-04-22 19:54:30,989 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 98809de8ceb9ac8f3bc773e820fd0e11/C of 98809de8ceb9ac8f3bc773e820fd0e11 into 5c0adfe2e64848d9859750bb8de2ff06(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:30,990 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:30,990 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., storeName=98809de8ceb9ac8f3bc773e820fd0e11/C, priority=11, startTime=1682193270770; duration=0sec 2023-04-22 19:54:30,990 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:31,408 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422b55ed6daf4944e7ba987160343dc444a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b55ed6daf4944e7ba987160343dc444a 2023-04-22 19:54:31,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:31,429 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:31,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:31,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193331446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:31,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:31,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193331549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:31,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:31,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43826 deadline: 1682193331649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:31,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:31,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43838 deadline: 1682193331655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:31,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:31,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43802 deadline: 1682193331656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:31,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:31,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193331751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:31,818 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=354, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/c6b0fe84152148e19424a23b4acf1019 2023-04-22 19:54:31,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:31,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43788 deadline: 1682193331835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:31,839 DEBUG [Thread-1000] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18218 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:54:32,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:32,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193332055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,229 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/228993d42b46498d9043a496538e3802 2023-04-22 19:54:32,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:32,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43766 deadline: 1682193332560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,641 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/6c1427fe485a4eacbc3b3ab09935f498 2023-04-22 19:54:32,649 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/c6b0fe84152148e19424a23b4acf1019 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c6b0fe84152148e19424a23b4acf1019 2023-04-22 19:54:32,653 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c6b0fe84152148e19424a23b4acf1019, entries=150, sequenceid=354, filesize=25.4 K 2023-04-22 19:54:32,654 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/228993d42b46498d9043a496538e3802 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/228993d42b46498d9043a496538e3802 2023-04-22 19:54:32,658 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/228993d42b46498d9043a496538e3802, entries=150, sequenceid=354, filesize=11.9 K 2023-04-22 19:54:32,659 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/6c1427fe485a4eacbc3b3ab09935f498 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/6c1427fe485a4eacbc3b3ab09935f498 2023-04-22 19:54:32,664 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/6c1427fe485a4eacbc3b3ab09935f498, entries=150, sequenceid=354, filesize=11.9 K 2023-04-22 19:54:32,665 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for 98809de8ceb9ac8f3bc773e820fd0e11 in 1715ms, sequenceid=354, compaction requested=false 2023-04-22 19:54:32,666 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:32,666 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:32,666 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:32,666 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:32,666 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,666 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:32,666 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:32,667 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,667 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:32,668 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:32,668 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:32,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:32,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1564207c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:32,670 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:32,670 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:32,672 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:32,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:32,672 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:32,672 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:32,672 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:32,672 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,672 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:32,674 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:32,674 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,674 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:32,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:32,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:32,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:32,677 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:32,677 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:32,677 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:32,677 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:32,677 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:32,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:32,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:32,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:32,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:32,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:32,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:32,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:32,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,681 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,683 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,683 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:32,683 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:32,683 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,683 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:32,683 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:32,684 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:32,684 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:32,684 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:32,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:32,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:32,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:32,684 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:32,684 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:32,684 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:32,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:32,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:32,784 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:32,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:32,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:32,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:32,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:32,788 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:32,788 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:32,788 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:32,788 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:32,788 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:32,788 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:32,791 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,791 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:32,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:32,791 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:32,791 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:32,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:32,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:32,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:32,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:32,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:32,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:32,793 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:32,793 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:32,793 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:32,793 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:32,793 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:32,793 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:32,793 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. started... 2023-04-22 19:54:32,793 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 19:54:32,793 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:32,794 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:32,794 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:32,794 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:32,794 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:32,794 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:32,807 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304224537845f187b4775bc36d16510fb3092 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224537845f187b4775bc36d16510fb3092 2023-04-22 19:54:32,813 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/86810c9b4d6c4976a21c20bb82afd838 2023-04-22 19:54:33,226 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/11c4be2a5cfa4a8f8283d8ac274de18e 2023-04-22 19:54:33,514 DEBUG [Thread-1013] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54451f11 to 127.0.0.1:53381 2023-04-22 19:54:33,514 DEBUG [Thread-1011] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0feb770d to 127.0.0.1:53381 2023-04-22 19:54:33,514 DEBUG [Thread-1013] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:33,514 DEBUG [Thread-1011] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:33,514 DEBUG [Thread-1015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:53381 2023-04-22 19:54:33,515 DEBUG [Thread-1015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:33,519 DEBUG [Thread-1017] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:53381 2023-04-22 19:54:33,519 DEBUG [Thread-1017] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:33,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:33,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. as already flushing 2023-04-22 19:54:33,573 DEBUG [Thread-1008] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69f15cb4 to 127.0.0.1:53381 2023-04-22 19:54:33,573 DEBUG [Thread-1008] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:33,659 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/8d1a6666fca344f0aa708803c8d14a6e 2023-04-22 19:54:33,663 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/86810c9b4d6c4976a21c20bb82afd838 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/86810c9b4d6c4976a21c20bb82afd838 2023-04-22 19:54:33,664 DEBUG [Thread-1006] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21265fcb to 127.0.0.1:53381 2023-04-22 19:54:33,664 DEBUG [Thread-1002] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33a47c05 to 127.0.0.1:53381 2023-04-22 19:54:33,664 DEBUG [Thread-1006] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:33,664 DEBUG [Thread-1002] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:33,668 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/86810c9b4d6c4976a21c20bb82afd838, entries=150, sequenceid=367, filesize=25.4 K 2023-04-22 19:54:33,669 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/11c4be2a5cfa4a8f8283d8ac274de18e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/11c4be2a5cfa4a8f8283d8ac274de18e 2023-04-22 19:54:33,671 DEBUG [Thread-1004] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1aa5eef5 to 127.0.0.1:53381 2023-04-22 19:54:33,671 DEBUG [Thread-1004] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:33,673 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/11c4be2a5cfa4a8f8283d8ac274de18e, entries=150, sequenceid=367, filesize=11.9 K 2023-04-22 19:54:33,674 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/8d1a6666fca344f0aa708803c8d14a6e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/8d1a6666fca344f0aa708803c8d14a6e 2023-04-22 19:54:33,678 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/8d1a6666fca344f0aa708803c8d14a6e, entries=150, sequenceid=367, filesize=11.9 K 2023-04-22 19:54:33,679 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=26.84 KB/27480 for 98809de8ceb9ac8f3bc773e820fd0e11 in 885ms, sequenceid=367, compaction requested=true 2023-04-22 19:54:33,679 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:33,679 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:33,680 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@160a685d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:33,682 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:33,682 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:33,685 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:33,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:33,686 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:33,686 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:33,686 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:33,686 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,686 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:33,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:33,687 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:33,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:33,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:33,689 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:33,689 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:33,689 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:33,689 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:33,689 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:33,691 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:33,691 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:33,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:33,691 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:33,691 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:33,692 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:33,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:33,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:33,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:33,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:33,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:33,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:33,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:33,694 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:33,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:33,695 DEBUG [Thread-1010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:33,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:33,695 DEBUG [Thread-1010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:33,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:33,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:33,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:33,795 DEBUG [Thread-1010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:33,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:37,585 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-22 19:54:41,925 DEBUG [Thread-1000] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00cbd61c to 127.0.0.1:53381 2023-04-22 19:54:41,925 DEBUG [Thread-1000] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:41,925 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 19:54:41,925 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 7 2023-04-22 19:54:41,925 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 65 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 5327 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 5278 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 2240 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 6716 rows 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 2233 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 6699 rows 2023-04-22 19:54:41,926 INFO [Listener at localhost/46665] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 19:54:41,926 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c16875c to 127.0.0.1:53381 2023-04-22 19:54:41,926 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:54:41,928 INFO [Listener at localhost/46665] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 19:54:41,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 19:54:41,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:41,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 19:54:41,934 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193281934"}]},"ts":"1682193281934"} 2023-04-22 19:54:41,935 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 19:54:41,941 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 19:54:41,942 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=98809de8ceb9ac8f3bc773e820fd0e11, UNASSIGN}] 2023-04-22 19:54:41,943 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=98809de8ceb9ac8f3bc773e820fd0e11, UNASSIGN 2023-04-22 19:54:41,943 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=98809de8ceb9ac8f3bc773e820fd0e11, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:41,944 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193281943"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193281943"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193281943"}]},"ts":"1682193281943"} 2023-04-22 19:54:41,944 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:54:42,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 19:54:42,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:42,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 98809de8ceb9ac8f3bc773e820fd0e11, disabling compactions & flushes 2023-04-22 19:54:42,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:42,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:42,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. after waiting 0 ms 2023-04-22 19:54:42,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:42,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 98809de8ceb9ac8f3bc773e820fd0e11 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-22 19:54:42,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=A 2023-04-22 19:54:42,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:42,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=B 2023-04-22 19:54:42,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:42,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98809de8ceb9ac8f3bc773e820fd0e11, store=C 2023-04-22 19:54:42,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:42,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 19:54:42,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304228dda97eb08314d03af8196049feebfee to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228dda97eb08314d03af8196049feebfee 2023-04-22 19:54:42,516 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=375, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/55237bf7e72d4d4b9df9750dc259d6e8 2023-04-22 19:54:42,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/9bf9cd86ff094397aaad7f61c0efe138 2023-04-22 19:54:42,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 19:54:42,934 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/b70031b6971e40068c390db28d6e587b 2023-04-22 19:54:42,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/A/55237bf7e72d4d4b9df9750dc259d6e8 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/55237bf7e72d4d4b9df9750dc259d6e8 2023-04-22 19:54:42,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/55237bf7e72d4d4b9df9750dc259d6e8, entries=150, sequenceid=375, filesize=25.4 K 2023-04-22 19:54:42,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/B/9bf9cd86ff094397aaad7f61c0efe138 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/9bf9cd86ff094397aaad7f61c0efe138 2023-04-22 19:54:42,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/9bf9cd86ff094397aaad7f61c0efe138, entries=150, sequenceid=375, filesize=11.9 K 2023-04-22 19:54:42,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/.tmp/C/b70031b6971e40068c390db28d6e587b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b70031b6971e40068c390db28d6e587b 2023-04-22 19:54:42,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b70031b6971e40068c390db28d6e587b, entries=150, sequenceid=375, filesize=11.9 K 2023-04-22 19:54:42,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 98809de8ceb9ac8f3bc773e820fd0e11 in 856ms, sequenceid=375, compaction requested=true 2023-04-22 19:54:42,973 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a7df4132e0ef42778a5a5b319ba66b58, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/e6292519f0614f908ede7776e062969c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d000b63b2b9041729ba40bb28ba127ca, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/8c9fbfea5829465c9cbf2c34f7b4c8cf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/cf6fe307f0d748e387d2003b67313de0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/7416387d57ca4ecb9f0f32489f72f912, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/151d60b73acd4934a7431131198d9679, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a1f73b43770f478f8259339b5e4aa4cd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/f3adc7f225af470281601abe0df8d30c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/3036225b621a4994b947ce0369f3728e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d743495e03d9449ca5feb5145299983e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/bb1950696c92409ca86d3fdb83f8cff5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d41d54d42c854164a073f895847b7c14, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/13bf49d6ccff4a999cb7ddf01408ef3e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d2d8939d76a043d1b6b2b9c1d7fce671, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/04a9185fbc734aa48fc38e0f5a823d6a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/3ae36414b8d247768c0b8acdf5a1e49d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/65bc9f1f21544e5abd6bae4e950244d6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/4d08d72274764d72b5bb39292e94efe4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/326ad17e3b7e450092fbd27da6ff5bd2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c19f73ff282f4d39b72e59d5829a44c3] to archive 2023-04-22 19:54:42,975 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:54:42,976 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a7df4132e0ef42778a5a5b319ba66b58 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a7df4132e0ef42778a5a5b319ba66b58 2023-04-22 19:54:42,978 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/e6292519f0614f908ede7776e062969c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/e6292519f0614f908ede7776e062969c 2023-04-22 19:54:42,978 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d000b63b2b9041729ba40bb28ba127ca to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d000b63b2b9041729ba40bb28ba127ca 2023-04-22 19:54:42,979 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/8c9fbfea5829465c9cbf2c34f7b4c8cf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/8c9fbfea5829465c9cbf2c34f7b4c8cf 2023-04-22 19:54:42,980 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/cf6fe307f0d748e387d2003b67313de0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/cf6fe307f0d748e387d2003b67313de0 2023-04-22 19:54:42,981 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/7416387d57ca4ecb9f0f32489f72f912 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/7416387d57ca4ecb9f0f32489f72f912 2023-04-22 19:54:42,982 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/151d60b73acd4934a7431131198d9679 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/151d60b73acd4934a7431131198d9679 2023-04-22 19:54:42,983 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a1f73b43770f478f8259339b5e4aa4cd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/a1f73b43770f478f8259339b5e4aa4cd 2023-04-22 19:54:42,984 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/f3adc7f225af470281601abe0df8d30c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/f3adc7f225af470281601abe0df8d30c 2023-04-22 19:54:42,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/3036225b621a4994b947ce0369f3728e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/3036225b621a4994b947ce0369f3728e 2023-04-22 19:54:42,986 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d743495e03d9449ca5feb5145299983e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d743495e03d9449ca5feb5145299983e 2023-04-22 19:54:42,986 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/bb1950696c92409ca86d3fdb83f8cff5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/bb1950696c92409ca86d3fdb83f8cff5 2023-04-22 19:54:42,987 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d41d54d42c854164a073f895847b7c14 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d41d54d42c854164a073f895847b7c14 2023-04-22 19:54:42,988 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/13bf49d6ccff4a999cb7ddf01408ef3e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/13bf49d6ccff4a999cb7ddf01408ef3e 2023-04-22 19:54:42,989 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d2d8939d76a043d1b6b2b9c1d7fce671 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/d2d8939d76a043d1b6b2b9c1d7fce671 2023-04-22 19:54:42,990 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/04a9185fbc734aa48fc38e0f5a823d6a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/04a9185fbc734aa48fc38e0f5a823d6a 2023-04-22 19:54:42,991 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/3ae36414b8d247768c0b8acdf5a1e49d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/3ae36414b8d247768c0b8acdf5a1e49d 2023-04-22 19:54:42,992 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/65bc9f1f21544e5abd6bae4e950244d6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/65bc9f1f21544e5abd6bae4e950244d6 2023-04-22 19:54:42,993 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/4d08d72274764d72b5bb39292e94efe4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/4d08d72274764d72b5bb39292e94efe4 2023-04-22 19:54:42,994 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/326ad17e3b7e450092fbd27da6ff5bd2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/326ad17e3b7e450092fbd27da6ff5bd2 2023-04-22 19:54:42,995 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c19f73ff282f4d39b72e59d5829a44c3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c19f73ff282f4d39b72e59d5829a44c3 2023-04-22 19:54:42,998 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dbaf3ad1fb564c498b444541cd959f4a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a28949d56462493a92d1253c90c0fc21, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/ee76b953fa5e466895f6b3e5f0c8991d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/7f953b5f72ee4b9490d2caf943b0e29c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/18a362f4211f4fd0816867f660002418, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/828ec8a108b049f081041e89dd5f10ac, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/19731fb4f43d478fa6ce3bd69b759a2c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/cd17d3e499b44bb1ae370569c2a19bc6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a06497dace1b4dc692b562c8d5b04b72, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/00a514c9d55d40579f61642f04a326f3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/49162f48167e4a9e89dafcd9698b5ccb, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/b27a4fc977cd43449cdbd801bbf27044, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/0547c166feae43f0be0d82890a0f536a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/3bd767519ee343a9bcfd2b819e3dbdca, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/8ee4bff1edb14a54ad866decff72fc7b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/9871dbb72af54ffc94e745ab88697172, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/818bc3c68fe445f99eb64a1aa4e7e4b2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/5c9aef63ec22406bb27b90078175943d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/ed018bba9dc6452085c8ce50878bfdd5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/56961b29c6de4ddf8c06fa024f9480ad, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dc7b2638579e4bc2a63e0005f6c84aee] to archive 2023-04-22 19:54:42,998 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:54:43,000 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dbaf3ad1fb564c498b444541cd959f4a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dbaf3ad1fb564c498b444541cd959f4a 2023-04-22 19:54:43,001 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a28949d56462493a92d1253c90c0fc21 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a28949d56462493a92d1253c90c0fc21 2023-04-22 19:54:43,002 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/ee76b953fa5e466895f6b3e5f0c8991d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/ee76b953fa5e466895f6b3e5f0c8991d 2023-04-22 19:54:43,003 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/7f953b5f72ee4b9490d2caf943b0e29c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/7f953b5f72ee4b9490d2caf943b0e29c 2023-04-22 19:54:43,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/18a362f4211f4fd0816867f660002418 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/18a362f4211f4fd0816867f660002418 2023-04-22 19:54:43,005 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/828ec8a108b049f081041e89dd5f10ac to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/828ec8a108b049f081041e89dd5f10ac 2023-04-22 19:54:43,006 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/19731fb4f43d478fa6ce3bd69b759a2c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/19731fb4f43d478fa6ce3bd69b759a2c 2023-04-22 19:54:43,007 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/cd17d3e499b44bb1ae370569c2a19bc6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/cd17d3e499b44bb1ae370569c2a19bc6 2023-04-22 19:54:43,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a06497dace1b4dc692b562c8d5b04b72 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/a06497dace1b4dc692b562c8d5b04b72 2023-04-22 19:54:43,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/00a514c9d55d40579f61642f04a326f3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/00a514c9d55d40579f61642f04a326f3 2023-04-22 19:54:43,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/49162f48167e4a9e89dafcd9698b5ccb to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/49162f48167e4a9e89dafcd9698b5ccb 2023-04-22 19:54:43,011 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/b27a4fc977cd43449cdbd801bbf27044 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/b27a4fc977cd43449cdbd801bbf27044 2023-04-22 19:54:43,012 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/0547c166feae43f0be0d82890a0f536a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/0547c166feae43f0be0d82890a0f536a 2023-04-22 19:54:43,013 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/3bd767519ee343a9bcfd2b819e3dbdca to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/3bd767519ee343a9bcfd2b819e3dbdca 2023-04-22 19:54:43,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/8ee4bff1edb14a54ad866decff72fc7b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/8ee4bff1edb14a54ad866decff72fc7b 2023-04-22 19:54:43,015 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/9871dbb72af54ffc94e745ab88697172 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/9871dbb72af54ffc94e745ab88697172 2023-04-22 19:54:43,016 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/818bc3c68fe445f99eb64a1aa4e7e4b2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/818bc3c68fe445f99eb64a1aa4e7e4b2 2023-04-22 19:54:43,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/5c9aef63ec22406bb27b90078175943d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/5c9aef63ec22406bb27b90078175943d 2023-04-22 19:54:43,018 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/ed018bba9dc6452085c8ce50878bfdd5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/ed018bba9dc6452085c8ce50878bfdd5 2023-04-22 19:54:43,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/56961b29c6de4ddf8c06fa024f9480ad to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/56961b29c6de4ddf8c06fa024f9480ad 2023-04-22 19:54:43,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dc7b2638579e4bc2a63e0005f6c84aee to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/dc7b2638579e4bc2a63e0005f6c84aee 2023-04-22 19:54:43,022 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/ceba049a52a643bd9ea73c60c4bb6a8d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b831ca77571744f9a1868b7027f75bd4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/944a65c174704c22aa8e6bc4746c55e4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/13c53ac0eca4471eba2259c957a6d2b0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4ea24463c1e5450e922e2c5218b932f4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/fba0a0f722a64b5098c6bc293c8f0f6b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/25ef82cdde3743d1ab6073fa0480b256, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/7885e54aaf894a8ca4d57011365cfccf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f08b6d0213dc48edb4115d28df50bc89, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/e862bb4f25f147288aa7232f884cf4b8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/c284536ba6d44e40ac0ae3a365c23a91, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b27e163339434456b291b30f86a1f8cc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/78b262f3830847b7951d42100311bae6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4097aeff7a6f42f0970377c9da035e76, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f8aca192fb06400bb3bc6f27e17297f3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/5068b45ebfa540fba081ebbcbd432370, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/2de212aefcc7411abb18bb6d4ba24e38, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/59c0367f75444e8188d89f4b92d282f7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/efc8e8fcc94b452b9f407c66867b02e5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/123f31625103455d9c78c74d88237fe3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/1c2c3f1283f542cfa32b4dab4329ccd2] to archive 2023-04-22 19:54:43,023 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:54:43,024 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/ceba049a52a643bd9ea73c60c4bb6a8d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/ceba049a52a643bd9ea73c60c4bb6a8d 2023-04-22 19:54:43,025 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b831ca77571744f9a1868b7027f75bd4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b831ca77571744f9a1868b7027f75bd4 2023-04-22 19:54:43,026 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/944a65c174704c22aa8e6bc4746c55e4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/944a65c174704c22aa8e6bc4746c55e4 2023-04-22 19:54:43,027 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/13c53ac0eca4471eba2259c957a6d2b0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/13c53ac0eca4471eba2259c957a6d2b0 2023-04-22 19:54:43,028 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4ea24463c1e5450e922e2c5218b932f4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4ea24463c1e5450e922e2c5218b932f4 2023-04-22 19:54:43,029 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/fba0a0f722a64b5098c6bc293c8f0f6b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/fba0a0f722a64b5098c6bc293c8f0f6b 2023-04-22 19:54:43,030 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/25ef82cdde3743d1ab6073fa0480b256 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/25ef82cdde3743d1ab6073fa0480b256 2023-04-22 19:54:43,031 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/7885e54aaf894a8ca4d57011365cfccf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/7885e54aaf894a8ca4d57011365cfccf 2023-04-22 19:54:43,033 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f08b6d0213dc48edb4115d28df50bc89 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f08b6d0213dc48edb4115d28df50bc89 2023-04-22 19:54:43,034 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/e862bb4f25f147288aa7232f884cf4b8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/e862bb4f25f147288aa7232f884cf4b8 2023-04-22 19:54:43,035 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/c284536ba6d44e40ac0ae3a365c23a91 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/c284536ba6d44e40ac0ae3a365c23a91 2023-04-22 19:54:43,036 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b27e163339434456b291b30f86a1f8cc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b27e163339434456b291b30f86a1f8cc 2023-04-22 19:54:43,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 19:54:43,037 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/78b262f3830847b7951d42100311bae6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/78b262f3830847b7951d42100311bae6 2023-04-22 19:54:43,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4097aeff7a6f42f0970377c9da035e76 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/4097aeff7a6f42f0970377c9da035e76 2023-04-22 19:54:43,039 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f8aca192fb06400bb3bc6f27e17297f3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/f8aca192fb06400bb3bc6f27e17297f3 2023-04-22 19:54:43,040 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/5068b45ebfa540fba081ebbcbd432370 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/5068b45ebfa540fba081ebbcbd432370 2023-04-22 19:54:43,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/2de212aefcc7411abb18bb6d4ba24e38 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/2de212aefcc7411abb18bb6d4ba24e38 2023-04-22 19:54:43,042 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/59c0367f75444e8188d89f4b92d282f7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/59c0367f75444e8188d89f4b92d282f7 2023-04-22 19:54:43,044 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/efc8e8fcc94b452b9f407c66867b02e5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/efc8e8fcc94b452b9f407c66867b02e5 2023-04-22 19:54:43,045 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/123f31625103455d9c78c74d88237fe3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/123f31625103455d9c78c74d88237fe3 2023-04-22 19:54:43,046 DEBUG [StoreCloser-TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/1c2c3f1283f542cfa32b4dab4329ccd2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/1c2c3f1283f542cfa32b4dab4329ccd2 2023-04-22 19:54:43,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/recovered.edits/378.seqid, newMaxSeqId=378, maxSeqId=4 2023-04-22 19:54:43,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11. 2023-04-22 19:54:43,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 98809de8ceb9ac8f3bc773e820fd0e11: 2023-04-22 19:54:43,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:43,054 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=98809de8ceb9ac8f3bc773e820fd0e11, regionState=CLOSED 2023-04-22 19:54:43,055 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193283054"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193283054"}]},"ts":"1682193283054"} 2023-04-22 19:54:43,057 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-22 19:54:43,057 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 98809de8ceb9ac8f3bc773e820fd0e11, server=jenkins-hbase4.apache.org,44791,1682193215766 in 1.1110 sec 2023-04-22 19:54:43,058 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-22 19:54:43,058 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=98809de8ceb9ac8f3bc773e820fd0e11, UNASSIGN in 1.1160 sec 2023-04-22 19:54:43,059 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193283059"}]},"ts":"1682193283059"} 2023-04-22 19:54:43,060 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 19:54:43,062 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 19:54:43,063 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1340 sec 2023-04-22 19:54:44,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 19:54:44,038 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-22 19:54:44,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 19:54:44,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:44,041 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:44,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-22 19:54:44,042 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:44,047 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:44,049 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/recovered.edits] 2023-04-22 19:54:44,051 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/55237bf7e72d4d4b9df9750dc259d6e8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/55237bf7e72d4d4b9df9750dc259d6e8 2023-04-22 19:54:44,053 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/86810c9b4d6c4976a21c20bb82afd838 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/86810c9b4d6c4976a21c20bb82afd838 2023-04-22 19:54:44,054 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c00ed7a63d424758b55f1745a060a8bd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c00ed7a63d424758b55f1745a060a8bd 2023-04-22 19:54:44,055 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c6b0fe84152148e19424a23b4acf1019 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/A/c6b0fe84152148e19424a23b4acf1019 2023-04-22 19:54:44,057 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/11c4be2a5cfa4a8f8283d8ac274de18e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/11c4be2a5cfa4a8f8283d8ac274de18e 2023-04-22 19:54:44,058 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/228993d42b46498d9043a496538e3802 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/228993d42b46498d9043a496538e3802 2023-04-22 19:54:44,059 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/8a9fa690b39c4d008c1aeea496711aef to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/8a9fa690b39c4d008c1aeea496711aef 2023-04-22 19:54:44,060 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/9bf9cd86ff094397aaad7f61c0efe138 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/B/9bf9cd86ff094397aaad7f61c0efe138 2023-04-22 19:54:44,062 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/5c0adfe2e64848d9859750bb8de2ff06 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/5c0adfe2e64848d9859750bb8de2ff06 2023-04-22 19:54:44,063 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/6c1427fe485a4eacbc3b3ab09935f498 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/6c1427fe485a4eacbc3b3ab09935f498 2023-04-22 19:54:44,065 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/8d1a6666fca344f0aa708803c8d14a6e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/8d1a6666fca344f0aa708803c8d14a6e 2023-04-22 19:54:44,066 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b70031b6971e40068c390db28d6e587b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/C/b70031b6971e40068c390db28d6e587b 2023-04-22 19:54:44,068 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/recovered.edits/378.seqid to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11/recovered.edits/378.seqid 2023-04-22 19:54:44,069 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/98809de8ceb9ac8f3bc773e820fd0e11 2023-04-22 19:54:44,069 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:54:44,070 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 19:54:44,070 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-22 19:54:44,074 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422172a736b0524433cad5fb26a18ec2fab to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422172a736b0524433cad5fb26a18ec2fab 2023-04-22 19:54:44,075 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422178dc211f92148fd8abde53924118e71 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422178dc211f92148fd8abde53924118e71 2023-04-22 19:54:44,077 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042225e60ce9e2d04f4cb2ffd4ba8cd79b28 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042225e60ce9e2d04f4cb2ffd4ba8cd79b28 2023-04-22 19:54:44,078 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223275e56c7e454d0c800a2cd4bf9093dc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223275e56c7e454d0c800a2cd4bf9093dc 2023-04-22 19:54:44,080 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422432adb4b0d974290a6389ad6d218081b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422432adb4b0d974290a6389ad6d218081b 2023-04-22 19:54:44,081 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224537845f187b4775bc36d16510fb3092 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224537845f187b4775bc36d16510fb3092 2023-04-22 19:54:44,082 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042256332e873add4414b8c335c5e82e09d2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042256332e873add4414b8c335c5e82e09d2 2023-04-22 19:54:44,083 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225ebced96946f450bacc4edffe4c12596 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225ebced96946f450bacc4edffe4c12596 2023-04-22 19:54:44,084 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042277eec7b5fc2542bca4bc7f7fc26ab8ad to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042277eec7b5fc2542bca4bc7f7fc26ab8ad 2023-04-22 19:54:44,085 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228dda97eb08314d03af8196049feebfee to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228dda97eb08314d03af8196049feebfee 2023-04-22 19:54:44,087 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304229134995d4eb2425ea43dcbd121fc4f08 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304229134995d4eb2425ea43dcbd121fc4f08 2023-04-22 19:54:44,088 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422924b57fd084c4a9087677b42e022a1e9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422924b57fd084c4a9087677b42e022a1e9 2023-04-22 19:54:44,089 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422948388b1ac484a41abb5bd8f98669101 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422948388b1ac484a41abb5bd8f98669101 2023-04-22 19:54:44,090 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a99dec55b74745f088f8aa9b8417eaf4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a99dec55b74745f088f8aa9b8417eaf4 2023-04-22 19:54:44,091 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b55ed6daf4944e7ba987160343dc444a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b55ed6daf4944e7ba987160343dc444a 2023-04-22 19:54:44,093 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c0543b14238a4a239a5becfb050073f6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c0543b14238a4a239a5becfb050073f6 2023-04-22 19:54:44,094 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c702b8f53efa4b80a72672b93e631413 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c702b8f53efa4b80a72672b93e631413 2023-04-22 19:54:44,095 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cc4ff1bc0bb04dd6a82b601e6b38e927 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cc4ff1bc0bb04dd6a82b601e6b38e927 2023-04-22 19:54:44,096 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d8ed4b72b75f4336953d35c7bf5d3776 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d8ed4b72b75f4336953d35c7bf5d3776 2023-04-22 19:54:44,097 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f86149da925d4adb9fffc72330326da6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f86149da925d4adb9fffc72330326da6 2023-04-22 19:54:44,098 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 19:54:44,100 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:44,102 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 19:54:44,104 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 19:54:44,105 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:44,105 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 19:54:44,105 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682193284105"}]},"ts":"9223372036854775807"} 2023-04-22 19:54:44,106 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 19:54:44,106 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 98809de8ceb9ac8f3bc773e820fd0e11, NAME => 'TestAcidGuarantees,,1682193250374.98809de8ceb9ac8f3bc773e820fd0e11.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 19:54:44,106 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 19:54:44,106 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682193284106"}]},"ts":"9223372036854775807"} 2023-04-22 19:54:44,108 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 19:54:44,110 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:44,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 72 msec 2023-04-22 19:54:44,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-22 19:54:44,143 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-22 19:54:44,155 INFO [Listener at localhost/46665] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 231), OpenFileDescriptor=503 (was 576), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=343 (was 413), ProcessCount=170 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=12539 (was 12668) 2023-04-22 19:54:44,166 INFO [Listener at localhost/46665] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=503, MaxFileDescriptor=60000, SystemLoadAverage=343, ProcessCount=170, AvailableMemoryMB=12538 2023-04-22 19:54:44,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 19:54:44,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 19:54:44,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 19:54:44,171 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 19:54:44,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-22 19:54:44,171 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 19:54:44,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 19:54:44,172 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393 2023-04-22 19:54:44,173 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393 empty. 2023-04-22 19:54:44,173 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393 2023-04-22 19:54:44,173 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:54:44,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 19:54:44,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 19:54:44,581 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 19:54:44,583 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => fba73082906c456896fddba31efd5393, NAME => 'TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp 2023-04-22 19:54:44,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 19:54:44,989 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:54:44,989 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing fba73082906c456896fddba31efd5393, disabling compactions & flushes 2023-04-22 19:54:44,989 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:44,989 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:44,989 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. after waiting 0 ms 2023-04-22 19:54:44,989 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:44,989 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:44,989 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:44,991 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 19:54:44,992 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193284992"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193284992"}]},"ts":"1682193284992"} 2023-04-22 19:54:44,993 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 19:54:44,994 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 19:54:44,994 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193284994"}]},"ts":"1682193284994"} 2023-04-22 19:54:44,994 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 19:54:44,998 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fba73082906c456896fddba31efd5393, ASSIGN}] 2023-04-22 19:54:45,000 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fba73082906c456896fddba31efd5393, ASSIGN 2023-04-22 19:54:45,001 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fba73082906c456896fddba31efd5393, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44791,1682193215766; forceNewPlan=false, retain=false 2023-04-22 19:54:45,152 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=fba73082906c456896fddba31efd5393, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:45,152 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193285152"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193285152"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193285152"}]},"ts":"1682193285152"} 2023-04-22 19:54:45,153 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:54:45,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 19:54:45,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:45,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fba73082906c456896fddba31efd5393, NAME => 'TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:54:45,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fba73082906c456896fddba31efd5393 2023-04-22 19:54:45,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:54:45,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fba73082906c456896fddba31efd5393 2023-04-22 19:54:45,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fba73082906c456896fddba31efd5393 2023-04-22 19:54:45,309 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fba73082906c456896fddba31efd5393 2023-04-22 19:54:45,310 DEBUG [StoreOpener-fba73082906c456896fddba31efd5393-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A 2023-04-22 19:54:45,310 DEBUG [StoreOpener-fba73082906c456896fddba31efd5393-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A 2023-04-22 19:54:45,311 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:54:45,311 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fba73082906c456896fddba31efd5393 columnFamilyName A 2023-04-22 19:54:45,312 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] regionserver.HStore(310): Store=fba73082906c456896fddba31efd5393/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:54:45,312 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fba73082906c456896fddba31efd5393 2023-04-22 19:54:45,313 DEBUG [StoreOpener-fba73082906c456896fddba31efd5393-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B 2023-04-22 19:54:45,313 DEBUG [StoreOpener-fba73082906c456896fddba31efd5393-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B 2023-04-22 19:54:45,313 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:54:45,313 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fba73082906c456896fddba31efd5393 columnFamilyName B 2023-04-22 19:54:45,314 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] regionserver.HStore(310): Store=fba73082906c456896fddba31efd5393/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:54:45,314 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fba73082906c456896fddba31efd5393 2023-04-22 19:54:45,315 DEBUG [StoreOpener-fba73082906c456896fddba31efd5393-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C 2023-04-22 19:54:45,315 DEBUG [StoreOpener-fba73082906c456896fddba31efd5393-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C 2023-04-22 19:54:45,315 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:54:45,315 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fba73082906c456896fddba31efd5393 columnFamilyName C 2023-04-22 19:54:45,316 INFO [StoreOpener-fba73082906c456896fddba31efd5393-1] regionserver.HStore(310): Store=fba73082906c456896fddba31efd5393/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:54:45,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:45,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393 2023-04-22 19:54:45,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393 2023-04-22 19:54:45,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 19:54:45,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fba73082906c456896fddba31efd5393 2023-04-22 19:54:45,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 19:54:45,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fba73082906c456896fddba31efd5393; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72733528, jitterRate=0.08381402492523193}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 19:54:45,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:45,322 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., pid=30, masterSystemTime=1682193285304 2023-04-22 19:54:45,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:45,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:45,324 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=fba73082906c456896fddba31efd5393, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:45,324 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193285323"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682193285323"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682193285323"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682193285323"}]},"ts":"1682193285323"} 2023-04-22 19:54:45,326 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-22 19:54:45,326 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 in 172 msec 2023-04-22 19:54:45,328 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-22 19:54:45,328 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fba73082906c456896fddba31efd5393, ASSIGN in 328 msec 2023-04-22 19:54:45,328 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 19:54:45,328 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193285328"}]},"ts":"1682193285328"} 2023-04-22 19:54:45,329 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 19:54:45,331 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 19:54:45,332 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1630 sec 2023-04-22 19:54:46,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 19:54:46,276 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-22 19:54:46,277 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x11545f45 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:46,282 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@458eac52, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:46,283 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:46,284 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:46,286 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 19:54:46,287 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49816, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 19:54:46,289 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x112921f2 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:46,293 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22b4eae5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:46,293 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x7ed1dde9 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:46,298 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262f4570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:46,298 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:46,302 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:46,302 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:46,306 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:46,306 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:46,310 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:46,310 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x1a24d617 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:46,315 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6919b2a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:46,315 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x104dbf4e to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:46,319 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ce5573e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:46,319 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x753156b2 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:46,323 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@87b3095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:46,323 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:46,327 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:46,327 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:54:46,332 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:54:46,338 DEBUG [hconnection-0x2ea4df52-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:46,339 DEBUG [hconnection-0xebdadfe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:46,340 DEBUG [hconnection-0x2609cc90-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:46,340 DEBUG [hconnection-0x623945af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:46,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:46,341 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:46,341 DEBUG [hconnection-0x417cfcfd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:46,341 DEBUG [hconnection-0x783449dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:46,341 DEBUG [hconnection-0x194fa333-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:46,342 DEBUG [hconnection-0x1998d18f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:46,342 DEBUG [hconnection-0x3c333ca0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:46,345 DEBUG [hconnection-0x7ad2e23e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:54:46,345 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:46,348 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:46,348 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:46,348 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:46,348 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:46,349 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59418, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:46,349 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:46,349 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59404, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:46,356 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59432, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:54:46,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:46,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:46,356 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:46,356 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:46,356 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:46,356 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:46,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:46,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:46,362 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:46,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:46,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:46,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:46,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:46,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:46,363 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:46,363 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:46,363 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:46,364 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:46,364 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:46,364 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:46,364 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:46,365 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-22 19:54:46,365 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,365 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,365 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:46,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:54:46,367 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:46,367 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:46,367 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:46,367 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:46,367 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:46,367 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:46,367 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:46,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193286409, max timestamp=1682193286415Num uniques 859; 2023-04-22 19:54:46,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:46,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193286409, max timestamp=1682193286415Num uniques 859; 2023-04-22 19:54:46,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193286409, max timestamp=1682193286415Num uniques 859; 2023-04-22 19:54:46,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:46,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:46,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193346426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193346426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193346427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59392 deadline: 1682193346427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682193346429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193346530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193346530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59392 deadline: 1682193346530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193346530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682193346532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193346734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193346735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59392 deadline: 1682193346735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193346735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:46,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682193346736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:46,790 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/fb70f378391f49cabd9a47051567ba82 2023-04-22 19:54:47,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:47,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193347037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:47,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59392 deadline: 1682193347037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:47,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193347038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:47,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193347038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:47,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682193347038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,204 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/f32ca17f508142aeab1dba6e8bee1302 2023-04-22 19:54:47,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:47,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193347540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:47,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193347542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:47,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59392 deadline: 1682193347542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:47,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193347543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:47,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682193347543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,614 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/065ecf64de204867a0ba0e97db7ffebd 2023-04-22 19:54:47,619 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/fb70f378391f49cabd9a47051567ba82 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/fb70f378391f49cabd9a47051567ba82 2023-04-22 19:54:47,623 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/fb70f378391f49cabd9a47051567ba82, entries=150, sequenceid=11, filesize=11.6 K 2023-04-22 19:54:47,624 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/f32ca17f508142aeab1dba6e8bee1302 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f32ca17f508142aeab1dba6e8bee1302 2023-04-22 19:54:47,628 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f32ca17f508142aeab1dba6e8bee1302, entries=100, sequenceid=11, filesize=9.3 K 2023-04-22 19:54:47,629 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/065ecf64de204867a0ba0e97db7ffebd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/065ecf64de204867a0ba0e97db7ffebd 2023-04-22 19:54:47,633 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/065ecf64de204867a0ba0e97db7ffebd, entries=100, sequenceid=11, filesize=9.3 K 2023-04-22 19:54:47,634 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fba73082906c456896fddba31efd5393 in 1270ms, sequenceid=11, compaction requested=false 2023-04-22 19:54:47,634 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-22 19:54:47,636 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:47,636 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:47,639 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,639 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:47,639 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:47,639 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:47,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:47,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:47,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55125017[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:47,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:47,642 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:47,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:47,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:47,645 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:47,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:47,648 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:47,648 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:47,648 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:47,648 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:47,648 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:47,650 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:47,650 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:47,650 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:47,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:47,650 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:47,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:47,652 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,655 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,655 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:47,655 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:47,655 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:47,655 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:47,655 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:47,655 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:47,655 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:47,655 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:47,655 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:47,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:47,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:47,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:47,656 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:47,656 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:47,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:47,756 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:47,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:47,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:47,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:47,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:47,759 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:47,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:47,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:47,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:47,760 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:47,760 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:47,761 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:47,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:47,762 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:47,762 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:47,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:47,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:47,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:47,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:47,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:47,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:47,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:47,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:47,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:47,763 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:47,763 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:47,763 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-22 19:54:47,764 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:47,764 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:47,764 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:47,764 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:47,764 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:47,764 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:48,175 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/6d451bfa530f453f95d3feac42c71d7d 2023-04-22 19:54:48,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:54:48,547 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:48,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:48,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193348551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:48,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:48,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:48,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59392 deadline: 1682193348552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:48,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682193348553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:48,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:48,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193348554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:48,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:48,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193348555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:48,582 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 19:54:48,588 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/083a25836f3f4a0bbb1545fc183a4693 2023-04-22 19:54:48,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:48,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193348655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:48,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:48,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193348657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:48,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:48,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193348659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:48,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:48,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193348858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:48,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:48,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193348860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:48,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:48,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193348863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,008 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/45bd57515df54889b735157b2686764d 2023-04-22 19:54:49,012 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/6d451bfa530f453f95d3feac42c71d7d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/6d451bfa530f453f95d3feac42c71d7d 2023-04-22 19:54:49,017 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/6d451bfa530f453f95d3feac42c71d7d, entries=150, sequenceid=48, filesize=11.6 K 2023-04-22 19:54:49,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/083a25836f3f4a0bbb1545fc183a4693 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/083a25836f3f4a0bbb1545fc183a4693 2023-04-22 19:54:49,022 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/083a25836f3f4a0bbb1545fc183a4693, entries=150, sequenceid=48, filesize=11.6 K 2023-04-22 19:54:49,023 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/45bd57515df54889b735157b2686764d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/45bd57515df54889b735157b2686764d 2023-04-22 19:54:49,027 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/45bd57515df54889b735157b2686764d, entries=150, sequenceid=48, filesize=11.6 K 2023-04-22 19:54:49,028 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for fba73082906c456896fddba31efd5393 in 1265ms, sequenceid=48, compaction requested=false 2023-04-22 19:54:49,028 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:49,028 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:49,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:49,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:49,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:49,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:49,030 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:49,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:49,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:49,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:49,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:49,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d606ed7[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:49,032 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:49,032 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:49,033 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:49,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:49,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:49,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:49,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:49,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:49,036 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,036 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:49,036 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:49,036 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:49,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:49,038 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:49,039 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:49,039 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:49,039 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:49,039 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:49,040 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:49,040 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:49,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:49,040 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:49,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:49,041 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:49,042 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:49,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:49,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:49,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:49,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:49,045 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:49,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:49,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:49,045 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:49,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,045 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:49,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:49,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:49,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:49,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:49,145 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:49,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:49,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:49,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:49,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:49,149 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:49,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:49,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:49,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:49,150 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:49,150 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:49,156 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:49,156 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:49,156 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,156 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:49,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:49,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:49,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:49,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:49,157 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:49,158 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:49,158 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 19:54:49,158 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:49,158 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:49,158 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:49,158 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:49,158 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:49,158 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:49,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:54:49,165 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:49,165 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/3c18d4ebcb3848a59d3a11113fca3a3c 2023-04-22 19:54:49,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193289177, max timestamp=1682193289185Num uniques 859; 2023-04-22 19:54:49,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:54:49,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193289177, max timestamp=1682193289185Num uniques 859; 2023-04-22 19:54:49,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:54:49,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193289177, max timestamp=1682193289185Num uniques 859; 2023-04-22 19:54:49,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:54:49,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193349203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193349204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193349205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193349307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193349307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193349308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193349510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193349510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193349512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,586 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/6fbaa414f9bf4a21986e5cdef7512c14 2023-04-22 19:54:49,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193349813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193349814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:49,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:49,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193349814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,000 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/d87705f5f79345e9a0a7a9db2a87aec5 2023-04-22 19:54:50,005 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/3c18d4ebcb3848a59d3a11113fca3a3c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3c18d4ebcb3848a59d3a11113fca3a3c 2023-04-22 19:54:50,009 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3c18d4ebcb3848a59d3a11113fca3a3c, entries=150, sequenceid=59, filesize=11.6 K 2023-04-22 19:54:50,010 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/6fbaa414f9bf4a21986e5cdef7512c14 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/6fbaa414f9bf4a21986e5cdef7512c14 2023-04-22 19:54:50,013 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/6fbaa414f9bf4a21986e5cdef7512c14, entries=150, sequenceid=59, filesize=11.6 K 2023-04-22 19:54:50,014 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/d87705f5f79345e9a0a7a9db2a87aec5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d87705f5f79345e9a0a7a9db2a87aec5 2023-04-22 19:54:50,017 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d87705f5f79345e9a0a7a9db2a87aec5, entries=150, sequenceid=59, filesize=11.6 K 2023-04-22 19:54:50,018 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fba73082906c456896fddba31efd5393 in 860ms, sequenceid=59, compaction requested=true 2023-04-22 19:54:50,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:50,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:50,020 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,020 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:50,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:50,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39b3d8c7[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:50,022 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:50,022 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:50,023 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:50,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:50,023 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:50,023 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:50,023 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:50,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:50,026 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:50,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:50,029 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:50,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:50,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:50,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:50,030 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:50,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:50,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:50,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:50,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:50,032 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:50,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:50,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:50,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:50,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:50,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:50,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:50,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:50,036 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:50,035 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:50,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:50,036 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:50,036 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:50,036 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:50,036 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:50,036 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:50,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:50,136 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:50,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:50,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:50,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:50,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:50,139 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:50,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:50,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:50,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:50,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:50,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:50,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,141 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:50,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:50,142 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,142 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:50,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:50,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:50,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:50,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:50,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:50,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:50,144 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:50,144 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:50,144 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-22 19:54:50,144 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:50,144 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:50,144 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:50,144 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:50,144 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:50,145 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:50,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:54:50,317 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:50,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193350323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193350324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193350326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193350427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193350427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193350429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,553 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/7b161d84514b46e59cb009ea183436f3 2023-04-22 19:54:50,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682193350557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,560 DEBUG [Thread-1678] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:54:50,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59392 deadline: 1682193350572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,575 DEBUG [Thread-1674] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:54:50,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193350630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193350631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193350634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193350932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193350934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:50,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193350936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:50,965 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/b9d5c38b77744c1a8b2b6529a6a6ffaf 2023-04-22 19:54:50,983 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/89c42b30759e4453b45db1c3aface021 2023-04-22 19:54:50,987 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/7b161d84514b46e59cb009ea183436f3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/7b161d84514b46e59cb009ea183436f3 2023-04-22 19:54:50,991 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/7b161d84514b46e59cb009ea183436f3, entries=150, sequenceid=93, filesize=11.6 K 2023-04-22 19:54:50,992 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/b9d5c38b77744c1a8b2b6529a6a6ffaf as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/b9d5c38b77744c1a8b2b6529a6a6ffaf 2023-04-22 19:54:50,999 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/b9d5c38b77744c1a8b2b6529a6a6ffaf, entries=150, sequenceid=93, filesize=11.6 K 2023-04-22 19:54:51,000 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/89c42b30759e4453b45db1c3aface021 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/89c42b30759e4453b45db1c3aface021 2023-04-22 19:54:51,005 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/89c42b30759e4453b45db1c3aface021, entries=150, sequenceid=93, filesize=11.6 K 2023-04-22 19:54:51,006 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for fba73082906c456896fddba31efd5393 in 862ms, sequenceid=93, compaction requested=true 2023-04-22 19:54:51,006 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:51,006 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,006 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:51,006 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:51,006 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,007 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:51,007 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:51,009 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,009 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,009 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,009 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:51,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c3e172c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:51,011 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:51,011 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,012 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:51,013 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,013 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:51,014 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:51,014 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,014 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,014 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:51,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:51,017 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:51,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:51,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:51,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:51,018 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:51,019 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,019 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,019 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,021 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,022 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,025 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,026 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,026 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,026 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,026 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:51,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:51,026 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,026 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,026 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:51,026 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,026 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:51,026 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,026 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,126 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:51,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:51,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:51,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:51,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:51,130 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:51,130 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,130 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:51,130 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:51,131 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,131 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,132 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,132 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,132 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,132 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:51,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:51,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:51,133 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:51,133 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,134 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:51,134 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:51,134 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:51,134 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,134 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:51,134 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 19:54:51,134 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:51,134 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:51,134 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:51,134 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:51,134 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:51,134 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:51,141 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/c3b16feea4f44bee9a1e5927ded2c41e 2023-04-22 19:54:51,151 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/2a303e70ac5746ec99eaae07934e4bc0 2023-04-22 19:54:51,161 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/1e0ea5bd6c15456aa314e8219d0e4165 2023-04-22 19:54:51,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/c3b16feea4f44bee9a1e5927ded2c41e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/c3b16feea4f44bee9a1e5927ded2c41e 2023-04-22 19:54:51,169 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/c3b16feea4f44bee9a1e5927ded2c41e, entries=150, sequenceid=104, filesize=11.6 K 2023-04-22 19:54:51,169 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/2a303e70ac5746ec99eaae07934e4bc0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/2a303e70ac5746ec99eaae07934e4bc0 2023-04-22 19:54:51,173 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/2a303e70ac5746ec99eaae07934e4bc0, entries=150, sequenceid=104, filesize=11.6 K 2023-04-22 19:54:51,174 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/1e0ea5bd6c15456aa314e8219d0e4165 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/1e0ea5bd6c15456aa314e8219d0e4165 2023-04-22 19:54:51,177 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/1e0ea5bd6c15456aa314e8219d0e4165, entries=150, sequenceid=104, filesize=11.6 K 2023-04-22 19:54:51,178 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for fba73082906c456896fddba31efd5393 in 44ms, sequenceid=104, compaction requested=true 2023-04-22 19:54:51,178 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:51,178 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:51,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:51,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:51,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:51,179 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,180 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,180 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b6b7254[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:51,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:51,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,182 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:51,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:51,183 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,183 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:51,185 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:51,188 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:51,188 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:51,188 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:51,188 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:51,188 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:51,190 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,190 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,190 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,191 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,192 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,195 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,195 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,195 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,195 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,195 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,195 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,195 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,195 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,195 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:51,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:51,195 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:51,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,196 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:51,196 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:51,296 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:51,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:51,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:51,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:51,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:51,299 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:51,299 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,299 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:51,299 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:51,300 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,300 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,301 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:51,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:51,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:51,303 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,303 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:51,303 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:51,303 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:51,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:51,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35f3b538[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:51,307 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:51,307 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,311 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:51,311 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,311 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,313 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:51,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:51,315 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:51,315 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:51,315 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:51,315 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:51,315 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:51,316 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,316 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,317 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,317 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,318 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:51,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:51,323 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,323 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,323 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,323 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:51,322 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,323 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:51,323 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,323 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,323 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:51,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,423 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:51,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:51,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:51,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:51,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:51,427 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:51,427 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,427 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:51,427 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:51,428 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,428 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,429 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,429 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,430 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,430 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:51,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:51,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:51,431 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,432 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:51,432 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:51,432 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,432 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:51,432 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:51,432 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,432 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:51,432 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:51,433 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,434 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,434 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:51,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@122136a1[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:51,435 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:51,436 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,437 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:51,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:51,439 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:51,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:51,442 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:51,442 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:51,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:51,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:51,443 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:51,444 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,444 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,444 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,445 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:54:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,446 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:54:51,446 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:51,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:51,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:51,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:51,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:51,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,450 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,450 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,450 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:51,450 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:51,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,450 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,451 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,451 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,451 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:51,451 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:51,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,451 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,451 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,451 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:51,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193351470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193291464, max timestamp=1682193291473Num uniques 859; 2023-04-22 19:54:51,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:54:51,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193291464, max timestamp=1682193291473Num uniques 859; 2023-04-22 19:54:51,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:54:51,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193291464, max timestamp=1682193291473Num uniques 859; 2023-04-22 19:54:51,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:54:51,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:51,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193351508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:51,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193351508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,551 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:51,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:51,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:51,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:51,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:51,555 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:51,555 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,555 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:51,555 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:51,555 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,555 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,556 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,556 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,557 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,557 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:51,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:51,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:51,558 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,558 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:51,558 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:51,558 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:51,560 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,560 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,561 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,561 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:51,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eaac4be[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:51,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:51,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,563 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,563 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:51,564 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:51,564 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:51,564 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,564 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:51,565 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:51,565 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,565 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,565 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:51,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:51,568 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:51,568 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:51,568 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:51,568 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:51,568 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:51,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,570 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,571 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:51,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:51,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,574 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:51,575 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:51,575 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:51,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,575 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,575 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:51,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193351577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:51,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193351612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:51,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193351612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,675 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:51,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:51,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:51,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:51,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:51,678 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:51,678 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,678 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:51,678 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:51,679 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,679 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,680 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,681 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:51,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:51,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:51,682 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,682 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:51,682 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:51,682 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,684 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:51,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e33eced[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:51,686 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:51,686 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,687 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:51,687 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,687 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:51,692 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:51,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:51,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:51,695 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:51,695 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:51,695 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:51,695 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:51,695 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:51,697 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,697 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,697 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,697 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,698 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:51,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,702 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:51,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:51,702 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:51,702 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:51,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:51,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193351779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,802 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:51,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:51,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:51,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:51,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:51,806 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:51,806 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,806 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:51,806 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:51,806 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,807 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,808 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,809 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,809 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:51,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:51,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:51,810 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:51,810 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:51,811 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,811 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:51,811 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:51,811 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:51,813 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,813 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,813 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,813 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5641cc66[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:51,814 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:51,814 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,815 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193351815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:51,816 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,816 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:51,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:51,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193351815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,817 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:51,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:51,820 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:51,820 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:51,820 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:51,820 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:51,820 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:51,823 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,823 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,823 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,823 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,827 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:51,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:51,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:51,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,828 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:51,828 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:51,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,855 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/deeefbc621084a54a28e92e272c03eed 2023-04-22 19:54:51,928 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:51,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:51,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:51,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:51,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:51,931 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:51,931 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,931 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:51,931 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:51,932 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,932 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,933 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,933 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,934 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,934 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:51,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:51,935 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,935 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:51,935 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:51,935 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:51,936 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:51,936 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,936 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:51,936 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:51,937 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dce6c31[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:51,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:51,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,949 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,950 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:51,950 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:51,950 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:51,950 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,950 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:51,951 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,951 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,951 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,952 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:51,954 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:51,954 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:51,954 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:51,954 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:51,954 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:51,955 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,956 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:51,956 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:51,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,956 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:51,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,957 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,959 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,959 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:51,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:51,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:51,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:51,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:51,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:51,960 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:51,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:51,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:51,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,960 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:51,960 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:51,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:51,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:51,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:52,060 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:52,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:52,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:52,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:52,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:52,064 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:52,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:52,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:52,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,071 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:52,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:52,071 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:52,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:52,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:52,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:52,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:52,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:52,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:52,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:52,073 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,073 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:52,073 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:52,073 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:52,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:52,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,074 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:52,074 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:52,075 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,075 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:52,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@509de02[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:52,077 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:52,077 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,078 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,078 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:52,078 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:52,078 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:52,078 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,078 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:52,079 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,079 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:52,079 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:52,079 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:52,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:52,082 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:52,082 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:52,082 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:52,082 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:52,082 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:52,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:52,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193352083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,089 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,089 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,089 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,089 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,095 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,095 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:52,095 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,095 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:52,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:52,095 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:52,095 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:52,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:52,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:52,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,096 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:52,096 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:52,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:52,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193352119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:52,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193352119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,196 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:52,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:52,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:52,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:52,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:52,200 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:52,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:52,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:52,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,201 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,201 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:52,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:52,202 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,202 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:52,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:52,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:52,203 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:52,203 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,203 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:52,203 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:52,203 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:52,203 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,204 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:52,204 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:52,204 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,205 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,206 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,206 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:52,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64bc0eca[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:52,207 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:52,207 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,209 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,209 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,209 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:52,209 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:52,209 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:52,209 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:52,211 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:52,213 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:52,213 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:52,213 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:52,213 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:52,213 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:52,215 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,215 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,215 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:52,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:52,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:52,215 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,216 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,219 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,219 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:52,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:52,219 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:52,219 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:52,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:52,219 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:52,219 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:52,219 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:52,219 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:52,219 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:52,219 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,219 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,219 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:52,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/cb77417aee184e238be4c4e23fdba735 2023-04-22 19:54:52,319 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:52,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:52,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:52,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:52,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:52,323 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:52,323 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,323 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:52,323 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:52,323 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,323 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,325 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:52,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:52,326 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,326 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:52,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:52,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:52,327 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:52,327 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,327 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:52,327 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:52,327 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:52,327 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,327 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:52,327 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:52,327 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:52,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:52,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:52,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:52,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,329 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,330 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,330 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:52,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cc6ccc7[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:52,339 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:52,339 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,340 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,340 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:52,340 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:52,340 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:52,340 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,340 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:52,341 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:52,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:52,344 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:52,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:52,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:52,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:52,344 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:52,352 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,352 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,352 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:52,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,356 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,356 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:52,356 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:52,356 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:52,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:52,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:52,356 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:52,356 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:52,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:52,357 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:52,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:52,357 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:52,357 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,357 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,357 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,357 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,457 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:52,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:52,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:52,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:52,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:52,460 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:52,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:52,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:52,461 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,461 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:52,462 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:52,463 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,463 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:52,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:52,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:52,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:52,464 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,464 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:52,464 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:52,464 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,466 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:52,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@257908a[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:52,468 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:52,468 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,470 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:52,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:52,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:52,471 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:52,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:52,474 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:52,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:52,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:52,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:52,474 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:52,475 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,475 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,476 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:52,476 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:52,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:52,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:52,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:52,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:52,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,483 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:52,484 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:52,484 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:52,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:52,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:52,584 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:52,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:52,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:52,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:52,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:52,588 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:52,588 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,588 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:52,588 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:52,588 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,588 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:52,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193352587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,589 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:52,589 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:52,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:52,590 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,590 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:52,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:52,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:52,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:52,591 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,591 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:52,592 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:52,592 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,592 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:52,592 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:52,592 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,592 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:52,592 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:52,593 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:52,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49e5b70c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:52,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:52,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,598 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,598 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:52,598 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:52,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,598 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:52,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:52,599 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:52,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:52,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:52,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:52,602 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:52,602 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:52,602 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:52,602 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:52,602 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:52,603 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,603 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:52,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,603 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:52,604 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,607 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,607 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,607 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,607 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:52,607 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:52,607 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:52,607 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:52,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:52,607 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:52,608 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:52,608 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:52,608 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:52,608 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:52,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:52,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193352625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:52,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193352626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/0b3ad2e0630a4db284f8bc6cdfdc1d5b 2023-04-22 19:54:52,692 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/deeefbc621084a54a28e92e272c03eed as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/deeefbc621084a54a28e92e272c03eed 2023-04-22 19:54:52,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/deeefbc621084a54a28e92e272c03eed, entries=150, sequenceid=117, filesize=11.6 K 2023-04-22 19:54:52,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/cb77417aee184e238be4c4e23fdba735 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/cb77417aee184e238be4c4e23fdba735 2023-04-22 19:54:52,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/cb77417aee184e238be4c4e23fdba735, entries=100, sequenceid=117, filesize=9.3 K 2023-04-22 19:54:52,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/0b3ad2e0630a4db284f8bc6cdfdc1d5b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/0b3ad2e0630a4db284f8bc6cdfdc1d5b 2023-04-22 19:54:52,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/0b3ad2e0630a4db284f8bc6cdfdc1d5b, entries=100, sequenceid=117, filesize=9.3 K 2023-04-22 19:54:52,707 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-22 19:54:52,708 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fba73082906c456896fddba31efd5393 in 1262ms, sequenceid=117, compaction requested=true 2023-04-22 19:54:52,708 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:52,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:52,708 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:52,708 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 19:54:52,708 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 19:54:52,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:52,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:52,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:52,710 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 66628 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 19:54:52,710 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/B is initiating minor compaction (all files) 2023-04-22 19:54:52,710 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/B in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,710 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71308 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 19:54:52,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:52,710 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/A is initiating minor compaction (all files) 2023-04-22 19:54:52,710 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f32ca17f508142aeab1dba6e8bee1302, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/083a25836f3f4a0bbb1545fc183a4693, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/6fbaa414f9bf4a21986e5cdef7512c14, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/b9d5c38b77744c1a8b2b6529a6a6ffaf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/2a303e70ac5746ec99eaae07934e4bc0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/cb77417aee184e238be4c4e23fdba735] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=65.1 K 2023-04-22 19:54:52,710 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/A in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,710 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/fb70f378391f49cabd9a47051567ba82, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/6d451bfa530f453f95d3feac42c71d7d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3c18d4ebcb3848a59d3a11113fca3a3c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/7b161d84514b46e59cb009ea183436f3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/c3b16feea4f44bee9a1e5927ded2c41e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/deeefbc621084a54a28e92e272c03eed] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=69.6 K 2023-04-22 19:54:52,711 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f32ca17f508142aeab1dba6e8bee1302, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682193286363 2023-04-22 19:54:52,711 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting fb70f378391f49cabd9a47051567ba82, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682193286361 2023-04-22 19:54:52,711 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 083a25836f3f4a0bbb1545fc183a4693, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682193286426 2023-04-22 19:54:52,711 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 6d451bfa530f453f95d3feac42c71d7d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682193286426 2023-04-22 19:54:52,711 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 6fbaa414f9bf4a21986e5cdef7512c14, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682193288549 2023-04-22 19:54:52,711 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 3c18d4ebcb3848a59d3a11113fca3a3c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682193288549 2023-04-22 19:54:52,712 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b9d5c38b77744c1a8b2b6529a6a6ffaf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682193289199 2023-04-22 19:54:52,712 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 7b161d84514b46e59cb009ea183436f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682193289199 2023-04-22 19:54:52,712 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 2a303e70ac5746ec99eaae07934e4bc0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1682193290319 2023-04-22 19:54:52,712 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c3b16feea4f44bee9a1e5927ded2c41e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1682193290319 2023-04-22 19:54:52,712 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting cb77417aee184e238be4c4e23fdba735, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682193291445 2023-04-22 19:54:52,712 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting deeefbc621084a54a28e92e272c03eed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682193291443 2023-04-22 19:54:52,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:52,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:52,724 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:52,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:52,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:52,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,727 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,727 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:52,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:52,727 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:52,727 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:52,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:52,732 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#B#compaction#233 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:52,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:52,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:52,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:52,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:52,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:52,747 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:52,747 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:52,747 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,747 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:52,747 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-22 19:54:52,747 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:52,747 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:52,747 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:52,748 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:52,748 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:52,748 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:52,755 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#A#compaction#234 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:52,772 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/1983138df74141dbb88f6b519d85950c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/1983138df74141dbb88f6b519d85950c 2023-04-22 19:54:52,778 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fba73082906c456896fddba31efd5393/B of fba73082906c456896fddba31efd5393 into 1983138df74141dbb88f6b519d85950c(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:52,778 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:52,778 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/B, priority=10, startTime=1682193292708; duration=0sec 2023-04-22 19:54:52,778 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:52,778 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 19:54:52,781 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 66628 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 19:54:52,781 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/C is initiating minor compaction (all files) 2023-04-22 19:54:52,781 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/C in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:52,781 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/065ecf64de204867a0ba0e97db7ffebd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/45bd57515df54889b735157b2686764d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d87705f5f79345e9a0a7a9db2a87aec5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/89c42b30759e4453b45db1c3aface021, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/1e0ea5bd6c15456aa314e8219d0e4165, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/0b3ad2e0630a4db284f8bc6cdfdc1d5b] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=65.1 K 2023-04-22 19:54:52,782 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 065ecf64de204867a0ba0e97db7ffebd, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682193286363 2023-04-22 19:54:52,782 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 45bd57515df54889b735157b2686764d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682193286426 2023-04-22 19:54:52,782 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting d87705f5f79345e9a0a7a9db2a87aec5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682193288549 2023-04-22 19:54:52,782 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 89c42b30759e4453b45db1c3aface021, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682193289199 2023-04-22 19:54:52,783 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1e0ea5bd6c15456aa314e8219d0e4165, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1682193290319 2023-04-22 19:54:52,783 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0b3ad2e0630a4db284f8bc6cdfdc1d5b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682193291445 2023-04-22 19:54:52,794 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#C#compaction#236 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:52,805 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/fbed117d8d094d6fbea15e7f47ff1f71 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/fbed117d8d094d6fbea15e7f47ff1f71 2023-04-22 19:54:52,811 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fba73082906c456896fddba31efd5393/C of fba73082906c456896fddba31efd5393 into fbed117d8d094d6fbea15e7f47ff1f71(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:52,811 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:52,811 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/C, priority=10, startTime=1682193292708; duration=0sec 2023-04-22 19:54:52,811 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:53,186 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/49b817bc73bf4fd8963df8fb1e0ca41a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/49b817bc73bf4fd8963df8fb1e0ca41a 2023-04-22 19:54:53,191 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fba73082906c456896fddba31efd5393/A of fba73082906c456896fddba31efd5393 into 49b817bc73bf4fd8963df8fb1e0ca41a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:53,191 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:53,191 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/A, priority=10, startTime=1682193292708; duration=0sec 2023-04-22 19:54:53,191 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:53,194 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/cef6ca67f07d4303bac97b8e2d737e63 2023-04-22 19:54:53,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:54:53,594 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:53,604 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/af4ce75135ef4f7cb1f5b9bbf56628b2 2023-04-22 19:54:53,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:53,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193353613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,615 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/d18af2ad37884c0b99a6e3a2c1d0fc06 2023-04-22 19:54:53,620 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/cef6ca67f07d4303bac97b8e2d737e63 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/cef6ca67f07d4303bac97b8e2d737e63 2023-04-22 19:54:53,624 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/cef6ca67f07d4303bac97b8e2d737e63, entries=150, sequenceid=152, filesize=11.8 K 2023-04-22 19:54:53,625 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/af4ce75135ef4f7cb1f5b9bbf56628b2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/af4ce75135ef4f7cb1f5b9bbf56628b2 2023-04-22 19:54:53,629 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/af4ce75135ef4f7cb1f5b9bbf56628b2, entries=150, sequenceid=152, filesize=11.8 K 2023-04-22 19:54:53,630 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/d18af2ad37884c0b99a6e3a2c1d0fc06 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d18af2ad37884c0b99a6e3a2c1d0fc06 2023-04-22 19:54:53,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:53,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193353632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,634 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d18af2ad37884c0b99a6e3a2c1d0fc06, entries=150, sequenceid=152, filesize=11.8 K 2023-04-22 19:54:53,634 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for fba73082906c456896fddba31efd5393 in 887ms, sequenceid=152, compaction requested=false 2023-04-22 19:54:53,635 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:53,635 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:53,636 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:53,637 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,637 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:54:53,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 19:54:53,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:53,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:53,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:53,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:53,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:53,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:53,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:53,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d2b3aa4[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:53,639 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:53,639 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,641 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:53,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:53,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:53,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:53,642 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:53,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:53,646 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:53,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:53,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:53,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:53,646 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:53,649 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,649 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:53,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,649 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:53,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:53,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:53,654 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:53,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:53,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:53,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:53,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:53,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:53,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,654 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:53,654 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,654 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:53,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:53,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:53,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193353687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:53,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193353715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,755 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:53,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:53,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:53,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:53,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:53,758 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:53,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:53,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:53,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:53,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,759 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:53,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:53,760 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,760 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:53,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:53,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:53,761 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:53,761 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:53,761 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:53,761 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:53,761 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:53,762 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:53,762 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:53,762 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:53,762 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:53,762 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:53,762 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:53,762 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,762 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:53,762 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:53,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:53,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:53,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:53,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47017732[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:53,765 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:53,765 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,772 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,772 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,772 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:53,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:53,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:53,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:53,774 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:53,774 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,774 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:53,774 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:53,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:53,776 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:53,776 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:53,776 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:53,776 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:53,776 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:53,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:53,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:53,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,778 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:53,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,779 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:53,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:53,781 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:53,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:53,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:53,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:53,781 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:53,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:53,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,781 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:53,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:53,782 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:53,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193353790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,882 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:53,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:53,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:53,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:53,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:53,885 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:53,885 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:53,885 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:53,885 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:53,885 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,886 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,890 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:53,890 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:53,891 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,891 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:53,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:53,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:53,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:53,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:53,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:53,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:53,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:53,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:53,892 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:53,892 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:53,892 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:53,892 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:53,897 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,897 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:53,897 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,897 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:53,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:53,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ec6c98[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:53,899 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:53,899 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,900 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:53,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:53,901 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:53,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,901 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:53,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:53,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:53,904 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:53,904 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:53,904 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:53,904 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:53,904 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:53,906 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,906 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:53,906 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:53,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:53,907 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:53,908 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:53,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:53,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:53,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:53,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:53,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:53,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:53,910 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:53,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:53,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:53,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:53,910 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:53,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:53,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:53,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193353919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:53,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:53,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193353995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,011 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:54,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:54,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:54,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:54,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:54,014 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:54,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:54,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:54,015 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,015 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,016 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,016 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,017 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,017 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:54,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:54,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:54,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:54,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:54,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:54,019 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,020 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,020 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,020 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:54,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d55e97e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:54,022 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:54,022 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,024 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,024 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,024 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:54,025 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,026 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:54,026 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,026 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:54,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:54,028 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:54,028 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:54,028 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:54,028 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:54,028 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:54,029 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,029 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,030 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,032 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,034 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:54,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:54,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,034 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:54,034 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:54,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,045 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/4187984e09904cee841d51913dcb97c0 2023-04-22 19:54:54,135 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:54,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:54,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:54,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:54,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:54,138 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:54,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:54,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:54,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,141 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,142 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,142 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:54,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:54,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:54,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:54,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:54,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:54,145 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,145 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,145 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,145 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:54,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@93f1e6a[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:54,147 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:54,148 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,149 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:54,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:54,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:54,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:54,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:54,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:54,151 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:54,153 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:54,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:54,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:54,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:54,153 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:54,155 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,155 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,155 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,156 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,157 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,159 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:54,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:54,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,160 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,160 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,160 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,160 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,160 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:54,160 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:54,160 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:54,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:54,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193354225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,260 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:54,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:54,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:54,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:54,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:54,263 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:54,263 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,263 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:54,263 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:54,264 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,264 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,265 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,265 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,265 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,265 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:54,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:54,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:54,266 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:54,266 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,266 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:54,267 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,267 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:54,267 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:54,267 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:54,268 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:54,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cb4098[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:54,270 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:54,270 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,272 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,272 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:54,272 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:54,272 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:54,272 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,272 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:54,273 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:54,276 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:54,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:54,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:54,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:54,276 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:54,277 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,277 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,277 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,278 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,279 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,281 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:54,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:54,282 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,282 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,282 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:54,282 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:54,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:54,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:54,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193354297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,382 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:54,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:54,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:54,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:54,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:54,385 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:54,385 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,385 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:54,385 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:54,386 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,386 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,388 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,388 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,388 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,388 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:54,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:54,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:54,390 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,390 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:54,390 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:54,390 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:54,391 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,392 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,392 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,392 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f596c15[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:54,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:54,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,394 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:54,395 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,395 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:54,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:54,398 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:54,398 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:54,398 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:54,398 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:54,398 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:54,401 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,401 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,401 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,401 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,402 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,404 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,405 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,405 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,405 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,405 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:54,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:54,405 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,405 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,405 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,405 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:54,405 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,405 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:54,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/3f00db7da25548938a24e3da15ef8294 2023-04-22 19:54:54,505 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:54,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:54,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:54,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:54,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:54,509 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:54,509 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,509 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:54,509 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:54,509 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,509 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,510 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,510 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,511 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,511 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:54,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:54,512 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:54,515 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,515 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,515 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,515 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d756099[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:54,517 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:54,517 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,519 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:54,519 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,519 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:54,520 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:54,523 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:54,523 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:54,523 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:54,523 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:54,523 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:54,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,526 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,527 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:54,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:54,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,529 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,529 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,529 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:54,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:54,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:54,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682193354570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,573 DEBUG [Thread-1678] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:54:54,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59392 deadline: 1682193354590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,593 DEBUG [Thread-1674] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:54:54,630 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:54,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:54,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:54,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:54,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:54,633 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:54,633 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,633 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:54,633 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:54,634 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,634 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,636 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,636 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,637 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,637 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:54,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:54,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:54,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:54,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:54,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:54,638 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,638 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:54,638 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:54,639 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,639 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:54,639 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:54,639 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,639 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:54,639 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:54,640 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,640 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,641 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,641 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:54,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:54,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c48ef13[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:54,642 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:54,642 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,643 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,643 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:54,643 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:54,643 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:54,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,644 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:54,648 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:54,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:54,651 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:54,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:54,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:54,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:54,651 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:54,652 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,652 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,652 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,653 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,654 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,656 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:54,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:54,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,657 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,657 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,657 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,657 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:54,657 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:54,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:54,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193354731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,757 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:54,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:54,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:54,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:54,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:54,761 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:54,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:54,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:54,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,761 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,763 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,763 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,763 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:54,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:54,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:54,765 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,765 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:54,765 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:54,765 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:54,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@398b382[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:54,769 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:54,770 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:54,771 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,771 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:54,772 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:54,775 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:54,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:54,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:54,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:54,775 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:54,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,778 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,779 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:54,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:54,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,784 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:54,784 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:54,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:54,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193354802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,884 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:54,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:54,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:54,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:54,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:54,888 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:54,888 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,888 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:54,888 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:54,888 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,888 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,890 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,890 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:54,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:54,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:54,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/18e2eb0e365243bab33370334da6a5a0 2023-04-22 19:54:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:54,892 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,892 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:54,892 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:54,892 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:54,894 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/4187984e09904cee841d51913dcb97c0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/4187984e09904cee841d51913dcb97c0 2023-04-22 19:54:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,896 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,896 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:54,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17355e20[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:54,897 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:54,897 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,898 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:54,898 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,898 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:54,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/4187984e09904cee841d51913dcb97c0, entries=150, sequenceid=167, filesize=11.8 K 2023-04-22 19:54:54,901 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:54,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,901 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,901 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:54,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/3f00db7da25548938a24e3da15ef8294 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3f00db7da25548938a24e3da15ef8294 2023-04-22 19:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:54,904 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:54,904 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:54,904 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:54,904 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:54,904 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:54,905 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,905 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:54,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,906 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:54,906 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,907 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3f00db7da25548938a24e3da15ef8294, entries=150, sequenceid=167, filesize=11.8 K 2023-04-22 19:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:54,908 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/18e2eb0e365243bab33370334da6a5a0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/18e2eb0e365243bab33370334da6a5a0 2023-04-22 19:54:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:54,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:54,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:54,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:54,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:54,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:54,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:54,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:54,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:54,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:54,910 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:54,910 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:54,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:54,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:54,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/18e2eb0e365243bab33370334da6a5a0, entries=150, sequenceid=167, filesize=11.8 K 2023-04-22 19:54:54,914 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fba73082906c456896fddba31efd5393 in 1276ms, sequenceid=167, compaction requested=true 2023-04-22 19:54:54,914 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:54,914 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:54,914 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:54,914 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:54,914 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:54,914 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:54,915 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:54,915 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:54,915 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/A is initiating minor compaction (all files) 2023-04-22 19:54:54,915 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/B is initiating minor compaction (all files) 2023-04-22 19:54:54,915 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/A in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,915 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/B in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,915 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/49b817bc73bf4fd8963df8fb1e0ca41a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/cef6ca67f07d4303bac97b8e2d737e63, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/4187984e09904cee841d51913dcb97c0] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=35.3 K 2023-04-22 19:54:54,915 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/1983138df74141dbb88f6b519d85950c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/af4ce75135ef4f7cb1f5b9bbf56628b2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3f00db7da25548938a24e3da15ef8294] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=35.3 K 2023-04-22 19:54:54,915 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 49b817bc73bf4fd8963df8fb1e0ca41a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682193290323 2023-04-22 19:54:54,916 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1983138df74141dbb88f6b519d85950c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682193290323 2023-04-22 19:54:54,916 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting cef6ca67f07d4303bac97b8e2d737e63, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1682193291502 2023-04-22 19:54:54,916 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting af4ce75135ef4f7cb1f5b9bbf56628b2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1682193291502 2023-04-22 19:54:54,916 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 4187984e09904cee841d51913dcb97c0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682193293603 2023-04-22 19:54:54,916 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 3f00db7da25548938a24e3da15ef8294, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682193293603 2023-04-22 19:54:54,922 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#A#compaction#242 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:54,922 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#B#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:54,934 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/12016cb4239a40c7a40d4a0a68974a1e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/12016cb4239a40c7a40d4a0a68974a1e 2023-04-22 19:54:54,938 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/B of fba73082906c456896fddba31efd5393 into 12016cb4239a40c7a40d4a0a68974a1e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:54,938 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:54,939 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/B, priority=13, startTime=1682193294914; duration=0sec 2023-04-22 19:54:54,939 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:54,939 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:54,940 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:54,940 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/C is initiating minor compaction (all files) 2023-04-22 19:54:54,940 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/C in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:54,940 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/fbed117d8d094d6fbea15e7f47ff1f71, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d18af2ad37884c0b99a6e3a2c1d0fc06, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/18e2eb0e365243bab33370334da6a5a0] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=35.3 K 2023-04-22 19:54:54,940 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting fbed117d8d094d6fbea15e7f47ff1f71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682193290323 2023-04-22 19:54:54,941 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting d18af2ad37884c0b99a6e3a2c1d0fc06, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1682193291502 2023-04-22 19:54:54,941 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 18e2eb0e365243bab33370334da6a5a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682193293603 2023-04-22 19:54:54,947 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#C#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:55,011 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:55,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:55,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:55,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:55,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:55,014 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:55,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:55,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:55,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:55,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,015 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:55,015 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:55,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:55,016 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,016 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:55,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:55,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:55,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:55,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:55,017 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:55,017 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:55,017 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:55,017 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:55,017 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:55,017 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:55,017 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:55,017 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 19:54:55,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:55,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:55,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:55,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:55,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:55,018 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:55,024 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/f8e925e60bb0481a89dd28dfef31ad9e 2023-04-22 19:54:55,336 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/2febaeacf5544b8097f7711fd8576a24 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/2febaeacf5544b8097f7711fd8576a24 2023-04-22 19:54:55,341 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/A of fba73082906c456896fddba31efd5393 into 2febaeacf5544b8097f7711fd8576a24(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:55,341 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:55,341 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/A, priority=13, startTime=1682193294914; duration=0sec 2023-04-22 19:54:55,342 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:55,355 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/ba15cbccd704476cb377893b3eccab66 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ba15cbccd704476cb377893b3eccab66 2023-04-22 19:54:55,359 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/C of fba73082906c456896fddba31efd5393 into ba15cbccd704476cb377893b3eccab66(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:55,359 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:55,359 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/C, priority=13, startTime=1682193294914; duration=0sec 2023-04-22 19:54:55,359 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:55,435 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/ea28156148764f759a569f035024447c 2023-04-22 19:54:55,445 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/ee44d66b3e424aec9ed5d5256cb2fdd0 2023-04-22 19:54:55,449 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/f8e925e60bb0481a89dd28dfef31ad9e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f8e925e60bb0481a89dd28dfef31ad9e 2023-04-22 19:54:55,453 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f8e925e60bb0481a89dd28dfef31ad9e, entries=150, sequenceid=188, filesize=11.8 K 2023-04-22 19:54:55,453 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/ea28156148764f759a569f035024447c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/ea28156148764f759a569f035024447c 2023-04-22 19:54:55,457 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/ea28156148764f759a569f035024447c, entries=150, sequenceid=188, filesize=11.8 K 2023-04-22 19:54:55,458 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/ee44d66b3e424aec9ed5d5256cb2fdd0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ee44d66b3e424aec9ed5d5256cb2fdd0 2023-04-22 19:54:55,464 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ee44d66b3e424aec9ed5d5256cb2fdd0, entries=150, sequenceid=188, filesize=11.8 K 2023-04-22 19:54:55,464 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for fba73082906c456896fddba31efd5393 in 447ms, sequenceid=188, compaction requested=false 2023-04-22 19:54:55,465 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:55,465 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:55,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:55,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:55,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@518285f7[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:55,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:55,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,471 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:55,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:55,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:55,471 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,471 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:55,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:55,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:55,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:55,476 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:55,476 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:55,476 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:55,476 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:55,476 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:55,478 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,478 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,478 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:55,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:55,478 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,486 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,486 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,486 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,486 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,486 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:55,486 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:55,486 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:55,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:55,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:55,486 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:55,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:55,486 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,486 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:55,487 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:55,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:55,587 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:55,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:55,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:55,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:55,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:55,599 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:55,599 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:55,599 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:55,599 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:55,599 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,599 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,600 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:55,600 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:55,601 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,601 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:55,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:55,602 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:55,602 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:55,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:55,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:55,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:55,603 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:55,603 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:55,603 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:55,603 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:55,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:55,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:55,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:55,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:55,604 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,604 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a69d8a3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:55,606 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:55,606 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,607 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:55,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:55,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:55,608 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,608 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:55,610 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:55,613 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:55,613 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:55,613 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:55,613 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:55,613 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:55,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:55,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,615 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,617 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:55,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:55,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:55,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:55,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:55,619 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:55,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,619 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:55,619 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:55,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:55,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:54:55,647 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:54:55,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:55,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:55,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:55,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:55,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:55,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:55,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193295670, max timestamp=1682193295686Num uniques 859; 2023-04-22 19:54:55,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193295670, max timestamp=1682193295686Num uniques 859; 2023-04-22 19:54:55,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193295670, max timestamp=1682193295686Num uniques 859; 2023-04-22 19:54:55,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:55,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:55,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:55,719 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:55,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:55,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:55,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:55,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:55,722 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:55,723 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:55,723 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:55,723 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:55,723 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,723 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,724 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,724 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:55,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:55,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:55,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:55,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:55,726 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:55,726 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:55,726 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:55,726 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:55,729 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:55,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c2b7ecf[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:55,731 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:55,731 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:55,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59418 deadline: 1682193355728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,732 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:55,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:55,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:55,732 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,732 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:55,734 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:55,734 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:55,734 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,734 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:55,736 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:55,736 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:55,736 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:55,736 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:55,736 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:55,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:55,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,738 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,739 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,742 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,742 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:55,742 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:55,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:55,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:55,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:55,742 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193355741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,742 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:55,742 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:55,742 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:55,742 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:55,743 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:55,743 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,743 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:55,743 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:55,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193355810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:55,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193355833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,843 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:55,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:55,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:55,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:55,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:55,847 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:55,847 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:55,847 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:55,847 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:55,847 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,847 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,848 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:55,848 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:55,849 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,849 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:55,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:55,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:55,850 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:55,850 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:55,850 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:55,850 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:55,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,853 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:55,855 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:55,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7152ed34[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:55,855 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,856 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:55,857 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,857 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:55,859 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:55,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:55,861 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:55,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:55,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:55,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:55,862 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:55,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:55,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,864 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,865 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:55,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:55,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:55,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:55,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:55,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,869 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:55,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,870 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:55,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:55,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,870 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:55,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,970 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:55,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:55,973 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:55,973 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:55,973 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:55,973 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:55,974 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,974 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,976 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:55,976 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:55,976 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,976 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:55,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:55,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:55,978 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:55,978 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:55,978 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:55,978 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:55,979 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,979 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,979 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,980 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:55,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@343a63d4[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:55,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:55,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,982 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,982 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:55,982 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:55,982 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:55,982 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,982 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:55,984 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:55,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:55,986 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:55,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:55,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:55,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:55,986 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:55,988 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,988 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,989 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:55,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:55,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:55,989 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:55,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:55,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:55,993 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:55,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:55,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:55,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:55,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:55,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:55,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:55,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:55,994 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:55,994 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:55,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:56,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:56,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193356035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,056 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/e267ab2fb428459c82bef2ffa3f68127 2023-04-22 19:54:56,094 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:56,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:56,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:56,097 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:56,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:56,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:56,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:56,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,098 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:56,098 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:56,099 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,099 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:56,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:56,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:56,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:56,100 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,100 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:56,100 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:56,100 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:56,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:56,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d20930[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:56,103 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:56,103 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,108 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:56,109 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,109 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:56,110 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,110 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:56,110 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,110 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:56,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:56,112 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:56,112 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:56,112 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:56,112 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:56,112 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:56,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:56,114 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,115 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,117 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,117 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,117 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,117 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,117 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:56,117 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:56,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:56,117 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:56,118 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:56,118 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,118 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:56,118 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:56,118 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:56,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:56,218 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:56,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:56,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:56,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:56,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:56,221 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:56,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:56,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:56,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:56,222 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,222 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,224 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,224 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:56,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:56,225 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,225 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:56,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:56,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:56,226 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,226 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:56,226 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:56,226 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:56,228 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:56,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6718dce3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:56,229 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:56,229 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,230 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,231 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:56,231 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:56,231 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:56,231 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,231 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:56,232 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:56,232 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:56,232 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,232 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:56,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:56,234 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:56,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:56,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:56,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:56,234 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:56,237 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,237 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,237 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:56,238 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,239 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,241 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:56,241 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:56,241 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:56,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:56,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:56,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:56,241 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,241 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:56,241 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,241 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,241 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,241 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,241 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,241 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:56,241 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:56,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:56,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:56,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193356338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,342 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:56,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:56,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:56,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:56,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:56,345 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:56,345 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:56,345 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:56,345 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:56,346 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,346 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,349 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:56,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:56,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:56,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:56,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:56,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:56,351 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,351 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:56,351 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:56,351 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:56,353 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:56,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@145a679[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:56,355 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:56,355 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,356 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:56,356 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,356 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:56,358 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:56,358 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:56,358 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,358 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:56,360 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:56,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:56,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:56,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:56,360 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:56,362 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,362 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,362 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:56,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,363 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,364 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,367 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:56,367 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:56,367 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:56,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:56,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:56,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:56,367 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:56,367 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,368 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:56,368 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:56,368 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:56,368 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,368 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,368 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,368 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,468 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:56,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:56,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:56,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:56,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:56,471 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:56,471 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:56,471 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:56,471 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:56,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/609b810bb775455cbc6ef4df55ab3312 2023-04-22 19:54:56,474 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:56,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:56,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:56,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:56,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:56,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:56,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:56,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:56,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:56,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:56,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:56,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:56,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:56,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:56,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:56,478 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:56,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66864146[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:56,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:56,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,481 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:56,481 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,482 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:56,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/3dc5e61e0cc0481a94b67bb85256e33a 2023-04-22 19:54:56,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:56,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:56,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:56,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:56,488 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:56,488 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:56,488 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:56,488 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:56,488 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:56,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/e267ab2fb428459c82bef2ffa3f68127 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/e267ab2fb428459c82bef2ffa3f68127 2023-04-22 19:54:56,493 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,493 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:56,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,493 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:56,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,496 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,498 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:56,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:56,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:56,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:56,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:56,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:56,499 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:56,499 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:56,499 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:56,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:56,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/e267ab2fb428459c82bef2ffa3f68127, entries=100, sequenceid=201, filesize=9.4 K 2023-04-22 19:54:56,506 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/609b810bb775455cbc6ef4df55ab3312 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/609b810bb775455cbc6ef4df55ab3312 2023-04-22 19:54:56,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/609b810bb775455cbc6ef4df55ab3312, entries=100, sequenceid=201, filesize=9.4 K 2023-04-22 19:54:56,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/3dc5e61e0cc0481a94b67bb85256e33a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3dc5e61e0cc0481a94b67bb85256e33a 2023-04-22 19:54:56,515 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3dc5e61e0cc0481a94b67bb85256e33a, entries=100, sequenceid=201, filesize=9.4 K 2023-04-22 19:54:56,515 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fba73082906c456896fddba31efd5393 in 868ms, sequenceid=201, compaction requested=true 2023-04-22 19:54:56,515 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:56,516 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:56,516 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:56,516 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:56,516 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:56,516 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:56,517 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:56,517 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/A is initiating minor compaction (all files) 2023-04-22 19:54:56,517 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/A in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,518 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/2febaeacf5544b8097f7711fd8576a24, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f8e925e60bb0481a89dd28dfef31ad9e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/e267ab2fb428459c82bef2ffa3f68127] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=33.2 K 2023-04-22 19:54:56,518 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:56,518 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/B is initiating minor compaction (all files) 2023-04-22 19:54:56,518 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/B in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,518 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/12016cb4239a40c7a40d4a0a68974a1e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/ea28156148764f759a569f035024447c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/609b810bb775455cbc6ef4df55ab3312] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=33.2 K 2023-04-22 19:54:56,518 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 2febaeacf5544b8097f7711fd8576a24, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682193293603 2023-04-22 19:54:56,519 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 12016cb4239a40c7a40d4a0a68974a1e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682193293603 2023-04-22 19:54:56,519 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting f8e925e60bb0481a89dd28dfef31ad9e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1682193293670 2023-04-22 19:54:56,519 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ea28156148764f759a569f035024447c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1682193293670 2023-04-22 19:54:56,519 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting e267ab2fb428459c82bef2ffa3f68127, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682193295638 2023-04-22 19:54:56,519 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 609b810bb775455cbc6ef4df55ab3312, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682193295638 2023-04-22 19:54:56,535 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#A#compaction#251 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:56,535 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#B#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:56,544 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/9c657455d4af48539137275563f924e2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/9c657455d4af48539137275563f924e2 2023-04-22 19:54:56,547 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/cccd46f8b7c941ac83cb4ee084804476 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/cccd46f8b7c941ac83cb4ee084804476 2023-04-22 19:54:56,548 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/A of fba73082906c456896fddba31efd5393 into 9c657455d4af48539137275563f924e2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:56,548 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:56,548 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/A, priority=13, startTime=1682193296516; duration=0sec 2023-04-22 19:54:56,548 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:56,548 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:56,549 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:56,549 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/C is initiating minor compaction (all files) 2023-04-22 19:54:56,550 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/C in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,550 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ba15cbccd704476cb377893b3eccab66, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ee44d66b3e424aec9ed5d5256cb2fdd0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3dc5e61e0cc0481a94b67bb85256e33a] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=33.2 K 2023-04-22 19:54:56,550 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting ba15cbccd704476cb377893b3eccab66, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682193293603 2023-04-22 19:54:56,550 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting ee44d66b3e424aec9ed5d5256cb2fdd0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1682193293670 2023-04-22 19:54:56,551 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 3dc5e61e0cc0481a94b67bb85256e33a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682193295638 2023-04-22 19:54:56,555 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/B of fba73082906c456896fddba31efd5393 into cccd46f8b7c941ac83cb4ee084804476(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:56,555 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:56,555 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/B, priority=13, startTime=1682193296516; duration=0sec 2023-04-22 19:54:56,555 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:56,558 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#C#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:56,599 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:56,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:56,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:56,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:56,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:56,603 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:56,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:56,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:56,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:56,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,604 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,604 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:56,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:56,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:56,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:56,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:56,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:56,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:56,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:56,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:56,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:56,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:56,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:56,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:56,606 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-22 19:54:56,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:56,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:56,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:56,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:56,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:56,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:56,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:54:56,843 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:56,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:56,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193356860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:56,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193356963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:56,970 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/506c6befd43f4b41be5f5fc3c38ac29d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/506c6befd43f4b41be5f5fc3c38ac29d 2023-04-22 19:54:56,974 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/C of fba73082906c456896fddba31efd5393 into 506c6befd43f4b41be5f5fc3c38ac29d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:56,974 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:56,974 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/C, priority=13, startTime=1682193296516; duration=0sec 2023-04-22 19:54:56,974 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:57,013 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/5fb76ae0b7374a3fa937231ea58592e4 2023-04-22 19:54:57,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:57,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193357166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,423 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/f74b47f9eddc47bb8fec30b77d8c44cb 2023-04-22 19:54:57,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:57,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193357470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:57,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193357756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,759 DEBUG [Thread-1682] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:54:57,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:57,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193357823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,826 DEBUG [Thread-1676] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:54:57,834 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/3c1fbbd332ef447482920c117df9fe7d 2023-04-22 19:54:57,839 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/5fb76ae0b7374a3fa937231ea58592e4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/5fb76ae0b7374a3fa937231ea58592e4 2023-04-22 19:54:57,842 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/5fb76ae0b7374a3fa937231ea58592e4, entries=150, sequenceid=240, filesize=11.8 K 2023-04-22 19:54:57,844 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/f74b47f9eddc47bb8fec30b77d8c44cb as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f74b47f9eddc47bb8fec30b77d8c44cb 2023-04-22 19:54:57,848 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f74b47f9eddc47bb8fec30b77d8c44cb, entries=150, sequenceid=240, filesize=11.8 K 2023-04-22 19:54:57,849 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/3c1fbbd332ef447482920c117df9fe7d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3c1fbbd332ef447482920c117df9fe7d 2023-04-22 19:54:57,852 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3c1fbbd332ef447482920c117df9fe7d, entries=150, sequenceid=240, filesize=11.8 K 2023-04-22 19:54:57,853 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for fba73082906c456896fddba31efd5393 in 1247ms, sequenceid=240, compaction requested=false 2023-04-22 19:54:57,853 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:57,853 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:57,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:57,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:57,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:57,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,856 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:57,857 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,857 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:57,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:57,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fd4f2f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:57,858 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:57,858 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,859 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:57,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:57,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:57,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:57,861 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:57,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:57,863 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:57,863 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:57,863 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:57,863 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:57,863 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:57,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:57,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:57,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:57,865 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,866 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:57,869 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:57,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:57,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:57,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:57,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:57,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:57,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:57,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,870 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:57,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:57,870 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:57,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:57,970 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:57,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:57,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:57,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:57,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:57,973 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:57,973 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:57,973 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:57,973 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:57,974 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,974 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:57,975 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,975 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:57,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:57,975 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,975 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:57,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:57,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:57,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:57,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:57,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:54:57,977 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:57,977 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 19:54:57,977 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:57,977 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:57,977 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:57,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:57,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:57,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:57,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:57,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:57,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:54:57,982 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:57,982 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,982 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,982 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:57,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:57,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27a374d7[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:57,984 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:57,984 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,986 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:57,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:57,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:57,987 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,987 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:57,988 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:57,988 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:57,988 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,988 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:57,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:57,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:57,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:57,991 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:57,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:57,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:57,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:57,991 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:57,992 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,992 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:57,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,992 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:57,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:57,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:57,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:57,993 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:57,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,994 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:57,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:57,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:57,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:57,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:57,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:57,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:57,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,996 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:57,997 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:57,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:57,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:57,997 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:57,997 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:57,997 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:57,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193298007, max timestamp=1682193298027Num uniques 859; 2023-04-22 19:54:58,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:58,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193298007, max timestamp=1682193298027Num uniques 859; 2023-04-22 19:54:58,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:58,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193298007, max timestamp=1682193298027Num uniques 859; 2023-04-22 19:54:58,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:54:58,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:58,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193358069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,097 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:58,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:58,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:58,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:58,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:58,100 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:58,100 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:58,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:58,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,103 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,103 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,104 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,104 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:58,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:58,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:58,105 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,105 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:58,105 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:58,106 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,106 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:58,106 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:58,106 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,106 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:58,106 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:58,107 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,107 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,108 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,108 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bf97834[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:58,109 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:58,109 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,111 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,111 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:58,111 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:58,111 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,111 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:58,111 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,112 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:58,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:58,114 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:58,115 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:58,115 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:58,115 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:58,115 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:58,117 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,117 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,117 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,119 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,122 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,122 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:58,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:58,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:58,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,122 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,122 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,123 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:58,122 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,123 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:58,123 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,123 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,123 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,123 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,123 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:58,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193358172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,223 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:58,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:58,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:58,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:58,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:58,226 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:58,226 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,226 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:58,226 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:58,227 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,227 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,228 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,228 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,228 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,228 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:58,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:58,230 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,230 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:58,230 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:58,230 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:58,232 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,232 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,232 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,232 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fa1947c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:58,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:58,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,236 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,237 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:58,237 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:58,237 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,237 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:58,237 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:58,238 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,238 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:58,238 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,238 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:58,241 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:58,241 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:58,241 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:58,241 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:58,241 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:58,243 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,243 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,243 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,244 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,245 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,247 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,247 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,247 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:58,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:58,247 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,247 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,247 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,247 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,248 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:58,248 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:58,248 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,248 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,248 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,348 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:58,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:58,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:58,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:58,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:58,351 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:58,351 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,351 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:58,351 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:58,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,354 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:58,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:58,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:58,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:58,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:58,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:58,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:58,355 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,355 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:58,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:58,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:58,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:58,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:58,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:58,357 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,357 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,357 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,357 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:58,359 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c92c90[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:58,359 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,360 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:58,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:58,360 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:58,360 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:58,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:58,368 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:58,368 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:58,368 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:58,368 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:58,368 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:58,369 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,369 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,369 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,370 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,371 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,373 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,373 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,373 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,373 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,373 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:58,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:58,373 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,373 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,374 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,374 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,374 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:58,374 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,374 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:58,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:58,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193358375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,386 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/547a0d5ef59a4d1f93d7ca7a541973b6 2023-04-22 19:54:58,474 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:58,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:58,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:58,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:58,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:58,477 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:58,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:58,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:58,478 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,478 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,479 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,479 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,479 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:58,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:58,480 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:58,480 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,480 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:58,481 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:58,481 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:58,481 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,481 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:58,481 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:58,481 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,481 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:58,481 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:58,481 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,481 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:58,481 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:58,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:58,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f70f3f2[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:58,485 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:58,485 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,486 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,486 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:58,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:58,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,488 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,488 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,488 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:58,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:58,490 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:58,490 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:58,491 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:58,491 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:58,491 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:58,492 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,492 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,492 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,493 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:58,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,498 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,498 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:58,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:58,498 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,498 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,498 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,498 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:58,498 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:58,598 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:58,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:58,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:58,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:58,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:58,602 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:58,602 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,602 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:58,602 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:58,602 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,604 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,604 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,604 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:58,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:58,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:58,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:58,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:58,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:58,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:58,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:58,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:58,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:58,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:58,607 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:58,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f0812f0[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:58,609 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:58,609 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,610 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:58,610 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,610 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:58,613 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,613 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:58,613 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,613 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:58,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:58,615 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:58,615 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:58,615 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:58,615 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:58,615 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:58,617 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,617 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,617 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,617 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,628 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:58,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:58,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,632 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,632 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,632 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,632 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,632 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:58,632 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:58,632 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:58,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:58,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193358678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,732 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:58,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:58,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:58,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:58,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:58,736 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:58,736 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,736 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:58,736 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:58,736 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,736 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,737 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,738 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,738 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:58,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:58,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:58,739 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:58,739 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,739 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:58,739 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:58,739 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:58,739 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,739 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:58,740 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:58,740 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:58,742 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:58,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c596d19[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:58,744 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:58,744 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,745 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,745 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:58,745 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:58,745 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:58,745 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,745 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:58,746 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:58,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:58,749 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:58,749 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:58,749 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:58,749 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:58,749 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:58,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,755 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,756 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,758 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,758 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,758 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,758 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,758 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:58,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:58,758 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,759 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,759 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,758 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,759 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:58,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,759 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:58,800 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/08d96db4c43b46b69ff5237fceeb89d4 2023-04-22 19:54:58,859 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:58,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:58,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:58,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:58,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:58,862 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:58,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:58,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:58,863 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,863 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,864 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,864 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,864 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,864 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:58,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:58,866 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,866 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:58,866 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:58,866 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:58,867 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@660e6fae[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:58,869 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:58,869 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,871 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,871 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,871 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:58,873 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:58,873 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,873 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,873 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:58,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:58,875 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:58,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:58,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:58,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:58,875 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:58,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,878 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,880 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,882 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:58,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:58,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,882 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:58,883 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:58,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:58,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:58,983 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:58,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:58,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:58,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:58,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:58,986 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:58,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:58,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:58,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,988 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:58,988 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:58,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:58,989 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,989 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:58,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:58,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:58,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:58,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:58,990 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:58,990 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:58,990 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:58,990 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:58,990 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:58,990 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,990 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:58,990 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:58,991 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:58,992 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,992 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,992 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,992 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:58,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e69fffe[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:58,994 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:58,994 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,995 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:58,995 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,995 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:58,997 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:58,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:58,999 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:58,999 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:58,999 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:58,999 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:58,999 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:59,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:59,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:59,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:59,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,009 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,011 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,011 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:59,011 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:59,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:59,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:59,011 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:59,011 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:59,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:59,011 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:59,012 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:59,012 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:59,012 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,012 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:59,012 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:59,012 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:59,112 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:59,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:59,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:59,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:59,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:59,116 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:59,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:59,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:59,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:59,117 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,117 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:59,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,118 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:59,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:59,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:59,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:59,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:59,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:59,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:59,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:59,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:59,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:59,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:59,120 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:59,120 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:59,120 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:59,120 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:54:59,121 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:54:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:54:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:54:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:54:59,122 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,122 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:59,122 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:59,122 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:54:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:59,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:54:59,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61f9ff51[Count = 0] remaining members to acquire global barrier 2023-04-22 19:54:59,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:54:59,124 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:59,126 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:59,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:59,126 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,126 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:54:59,126 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:54:59,126 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:54:59,126 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:54:59,127 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:54:59,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:59,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:59,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:59,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:54:59,130 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:54:59,130 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:54:59,130 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:54:59,130 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:54:59,130 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:54:59,131 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,131 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,132 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:59,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:54:59,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:59,132 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:54:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,134 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:54:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:54:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:59,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:59,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:54:59,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:54:59,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:54:59,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:59,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:54:59,140 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:54:59,140 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:54:59,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:54:59,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:59,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:54:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:54:59,140 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,140 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:59,140 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:54:59,140 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:54:59,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193359183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/b3577bdb50fc4934ad8af629bde1163e 2023-04-22 19:54:59,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/547a0d5ef59a4d1f93d7ca7a541973b6 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/547a0d5ef59a4d1f93d7ca7a541973b6 2023-04-22 19:54:59,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/547a0d5ef59a4d1f93d7ca7a541973b6, entries=200, sequenceid=252, filesize=14.1 K 2023-04-22 19:54:59,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/08d96db4c43b46b69ff5237fceeb89d4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/08d96db4c43b46b69ff5237fceeb89d4 2023-04-22 19:54:59,226 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/08d96db4c43b46b69ff5237fceeb89d4, entries=150, sequenceid=252, filesize=11.8 K 2023-04-22 19:54:59,227 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/b3577bdb50fc4934ad8af629bde1163e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/b3577bdb50fc4934ad8af629bde1163e 2023-04-22 19:54:59,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/b3577bdb50fc4934ad8af629bde1163e, entries=150, sequenceid=252, filesize=11.8 K 2023-04-22 19:54:59,230 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fba73082906c456896fddba31efd5393 in 1253ms, sequenceid=252, compaction requested=true 2023-04-22 19:54:59,231 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:59,231 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:59,231 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:59,231 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:59,231 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:59,231 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:59,232 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:59,232 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38908 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:59,232 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/B is initiating minor compaction (all files) 2023-04-22 19:54:59,232 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/A is initiating minor compaction (all files) 2023-04-22 19:54:59,232 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/B in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:59,232 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/A in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:59,232 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/cccd46f8b7c941ac83cb4ee084804476, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f74b47f9eddc47bb8fec30b77d8c44cb, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/08d96db4c43b46b69ff5237fceeb89d4] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=35.7 K 2023-04-22 19:54:59,232 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/9c657455d4af48539137275563f924e2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/5fb76ae0b7374a3fa937231ea58592e4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/547a0d5ef59a4d1f93d7ca7a541973b6] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=38.0 K 2023-04-22 19:54:59,233 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting cccd46f8b7c941ac83cb4ee084804476, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682193293670 2023-04-22 19:54:59,233 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 9c657455d4af48539137275563f924e2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682193293670 2023-04-22 19:54:59,233 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f74b47f9eddc47bb8fec30b77d8c44cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682193295721 2023-04-22 19:54:59,233 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 5fb76ae0b7374a3fa937231ea58592e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682193295721 2023-04-22 19:54:59,233 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 08d96db4c43b46b69ff5237fceeb89d4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682193296853 2023-04-22 19:54:59,233 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 547a0d5ef59a4d1f93d7ca7a541973b6, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682193296853 2023-04-22 19:54:59,240 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:54:59,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:54:59,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:54:59,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:54:59,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:54:59,246 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:54:59,246 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:59,246 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:54:59,246 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:54:59,247 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#B#compaction#260 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:59,247 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#A#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:59,248 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,248 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:59,250 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:54:59,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:54:59,250 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,251 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:54:59,251 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:54:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:59,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:54:59,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:54:59,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:54:59,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:54:59,258 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:54:59,258 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:54:59,258 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:54:59,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:54:59,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:54:59,260 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:59,260 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:54:59,260 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-22 19:54:59,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:54:59,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:59,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:54:59,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:59,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:54:59,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:54:59,275 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/bcfb7bcdae6e4e3d9976ddaf1a81c572 2023-04-22 19:54:59,656 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/240880a287c046eab0e3c90b7765857c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/240880a287c046eab0e3c90b7765857c 2023-04-22 19:54:59,660 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/B of fba73082906c456896fddba31efd5393 into 240880a287c046eab0e3c90b7765857c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:59,660 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:59,661 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/B, priority=13, startTime=1682193299231; duration=0sec 2023-04-22 19:54:59,661 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:54:59,661 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:54:59,662 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:54:59,662 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/C is initiating minor compaction (all files) 2023-04-22 19:54:59,662 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/C in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:54:59,662 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/506c6befd43f4b41be5f5fc3c38ac29d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3c1fbbd332ef447482920c117df9fe7d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/b3577bdb50fc4934ad8af629bde1163e] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=35.7 K 2023-04-22 19:54:59,663 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 506c6befd43f4b41be5f5fc3c38ac29d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682193293670 2023-04-22 19:54:59,663 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 3c1fbbd332ef447482920c117df9fe7d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682193295721 2023-04-22 19:54:59,663 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b3577bdb50fc4934ad8af629bde1163e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682193296853 2023-04-22 19:54:59,669 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#C#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:54:59,670 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/c151c83b980c4d64a066e9a1247d11a7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/c151c83b980c4d64a066e9a1247d11a7 2023-04-22 19:54:59,675 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/A of fba73082906c456896fddba31efd5393 into c151c83b980c4d64a066e9a1247d11a7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:54:59,675 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:54:59,675 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/A, priority=13, startTime=1682193299231; duration=0sec 2023-04-22 19:54:59,675 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:54:59,692 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/f38690805dd641ef81248410cdea033f 2023-04-22 19:55:00,079 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/b9f35a1b4009405c85cf066ed6126dab as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/b9f35a1b4009405c85cf066ed6126dab 2023-04-22 19:55:00,083 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/C of fba73082906c456896fddba31efd5393 into b9f35a1b4009405c85cf066ed6126dab(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:00,083 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:00,083 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/C, priority=13, startTime=1682193299231; duration=0sec 2023-04-22 19:55:00,084 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:00,106 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/00c6e812433c4ba2b9b8a242310e6a6d 2023-04-22 19:55:00,119 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/bcfb7bcdae6e4e3d9976ddaf1a81c572 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/bcfb7bcdae6e4e3d9976ddaf1a81c572 2023-04-22 19:55:00,125 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/bcfb7bcdae6e4e3d9976ddaf1a81c572, entries=150, sequenceid=288, filesize=11.9 K 2023-04-22 19:55:00,126 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/f38690805dd641ef81248410cdea033f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f38690805dd641ef81248410cdea033f 2023-04-22 19:55:00,133 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f38690805dd641ef81248410cdea033f, entries=150, sequenceid=288, filesize=11.9 K 2023-04-22 19:55:00,137 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/00c6e812433c4ba2b9b8a242310e6a6d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/00c6e812433c4ba2b9b8a242310e6a6d 2023-04-22 19:55:00,142 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/00c6e812433c4ba2b9b8a242310e6a6d, entries=150, sequenceid=288, filesize=11.9 K 2023-04-22 19:55:00,143 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for fba73082906c456896fddba31efd5393 in 883ms, sequenceid=288, compaction requested=false 2023-04-22 19:55:00,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:00,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:00,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:00,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:00,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,145 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:00,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@361f67d2[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:00,148 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:00,148 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,149 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:00,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:00,151 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:00,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:00,154 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:00,154 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:00,154 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:00,155 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:00,155 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:00,157 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,157 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,157 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,157 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,159 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:00,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:00,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:00,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,162 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:00,162 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:00,162 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:55:00,206 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:55:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:55:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:55:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:55:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:00,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/8d4b421998d448e6bfa5a9aeb794dcb5 2023-04-22 19:55:00,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193300246, max timestamp=1682193300254Num uniques 859; 2023-04-22 19:55:00,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193300246, max timestamp=1682193300254Num uniques 859; 2023-04-22 19:55:00,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193300246, max timestamp=1682193300254Num uniques 859; 2023-04-22 19:55:00,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:00,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:00,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:00,263 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:00,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:00,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:00,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:00,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:00,266 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:00,266 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,266 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:00,266 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:00,266 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,267 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,268 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,268 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,268 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,268 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:00,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:00,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:00,270 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,270 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:00,270 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:00,270 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,271 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:00,271 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,273 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:00,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23f91da0[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:00,274 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:00,274 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,276 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:00,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:00,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:00,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:00,277 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:00,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:00,280 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:00,280 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:00,280 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:00,280 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:00,280 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:00,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,282 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,284 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,288 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,288 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,288 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:00,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:00,288 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,288 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,289 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:00,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,289 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:00,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:00,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193360315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,389 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:00,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:00,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:00,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:00,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:00,392 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:00,392 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,392 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:00,392 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:00,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,394 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,394 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,394 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,394 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:00,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:00,395 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:00,395 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,395 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:00,395 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:00,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,398 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,398 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:00,399 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24e802e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:00,399 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,400 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:00,401 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,401 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:00,401 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:00,403 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,403 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,403 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,403 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:00,405 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:00,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:00,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:00,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:00,405 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:00,407 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,407 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,407 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,408 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,409 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,410 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,411 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:00,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:00,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:00,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,411 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,411 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:00,411 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:00,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:00,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193360419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,511 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:00,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:00,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:00,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:00,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:00,515 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:00,515 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,515 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:00,515 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:00,515 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,515 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,520 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:00,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:00,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:00,521 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:00,521 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,521 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:00,521 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:00,521 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:00,521 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,521 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:00,521 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:00,521 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,521 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:00,521 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:00,522 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,522 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:00,522 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,523 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:00,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ce3f441[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:00,525 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:00,525 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,526 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,526 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:00,526 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:00,526 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:00,526 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,526 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:00,527 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:00,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:00,529 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:00,529 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:00,530 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:00,530 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:00,530 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:00,532 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,532 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,532 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,533 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,534 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,536 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,536 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,536 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:00,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,536 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,536 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,536 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,537 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:00,537 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:00,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:00,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,537 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:00,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193360622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/3bed3ed17235463782f3dc547ea49637 2023-04-22 19:55:00,637 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:00,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:00,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:00,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:00,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:00,641 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:00,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:00,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:00,642 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,642 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,643 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,644 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,644 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:00,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:00,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:00,645 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:00,645 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,645 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:00,645 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:00,645 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:00,645 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,645 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:00,645 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:00,646 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,646 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:00,646 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:00,646 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,646 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:00,646 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:00,647 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,647 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:00,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@618f37f8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:00,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:00,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,651 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,651 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:00,651 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:00,651 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:00,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,652 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:00,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:00,655 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:00,655 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:00,655 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:00,655 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:00,655 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:00,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,656 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,657 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,658 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,661 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,661 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,661 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,661 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,661 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:00,661 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:00,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,661 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,661 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,661 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,661 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,661 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:00,661 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:00,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,761 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:00,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:00,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:00,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:00,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:00,765 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:00,765 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,765 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:00,765 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:00,766 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,766 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,768 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,768 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,768 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:00,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:00,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:00,770 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:00,770 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:00,770 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,770 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:00,770 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:00,770 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,770 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:00,770 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:00,770 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,770 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:00,770 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:00,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,771 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:00,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a702274[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:00,773 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:00,773 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,774 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,774 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:00,774 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:00,775 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:00,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:00,776 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:00,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:00,778 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:00,778 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:00,778 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:00,778 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:00,778 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:00,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,783 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:00,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:00,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,785 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:00,785 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:00,785 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,786 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,786 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,885 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:00,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:00,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:00,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:00,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:00,889 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:00,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:00,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:00,890 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,890 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,891 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,891 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,891 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,891 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:00,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:00,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:00,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:00,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:00,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:00,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:00,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:00,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:00,893 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,893 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:00,893 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:00,893 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:00,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:00,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:00,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:00,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:00,898 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:00,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c499ce4[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:00,899 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:00,899 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,900 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,901 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:00,901 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:00,901 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:00,901 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,901 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:00,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:00,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,902 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:00,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:00,904 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:00,905 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:00,905 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:00,905 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:00,905 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:00,906 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:00,906 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,906 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,906 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:00,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,908 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,910 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:00,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:00,911 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:00,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:00,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:00,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:00,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:00,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:00,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:00,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:00,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:00,911 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:00,911 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:00,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:00,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:00,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193360925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,011 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:01,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:01,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:01,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:01,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:01,015 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:01,015 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:01,015 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:01,015 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:01,015 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:01,015 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:01,017 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:01,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:01,017 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:01,017 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,017 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:01,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:01,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:01,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:01,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:01,019 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:01,019 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:01,019 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:01,019 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:01,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:01,021 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:01,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:01,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:01,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:01,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:01,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:01,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:01,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fbb2ed0[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:01,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:01,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:01,024 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:01,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:01,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:01,024 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,024 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:01,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,031 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:01,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:01,031 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:01,032 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:01,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:01,034 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:01,034 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:01,034 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:01,034 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:01,034 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:01,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:01,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:01,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:01,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:01,036 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:01,037 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/a274331d29574c158a4e67f2730a1d52 2023-04-22 19:55:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,040 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,040 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:01,040 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:01,040 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:01,040 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:01,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:01,040 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:01,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:01,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:01,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:01,041 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:01,041 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:01,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:01,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:01,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:01,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:01,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/8d4b421998d448e6bfa5a9aeb794dcb5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/8d4b421998d448e6bfa5a9aeb794dcb5 2023-04-22 19:55:01,046 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/8d4b421998d448e6bfa5a9aeb794dcb5, entries=150, sequenceid=301, filesize=11.9 K 2023-04-22 19:55:01,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/3bed3ed17235463782f3dc547ea49637 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3bed3ed17235463782f3dc547ea49637 2023-04-22 19:55:01,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3bed3ed17235463782f3dc547ea49637, entries=150, sequenceid=301, filesize=11.9 K 2023-04-22 19:55:01,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/a274331d29574c158a4e67f2730a1d52 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/a274331d29574c158a4e67f2730a1d52 2023-04-22 19:55:01,055 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/a274331d29574c158a4e67f2730a1d52, entries=150, sequenceid=301, filesize=11.9 K 2023-04-22 19:55:01,056 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fba73082906c456896fddba31efd5393 in 850ms, sequenceid=301, compaction requested=true 2023-04-22 19:55:01,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:01,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:01,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:01,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:01,056 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:55:01,056 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:55:01,058 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:55:01,058 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/B is initiating minor compaction (all files) 2023-04-22 19:55:01,058 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/B in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:01,058 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/240880a287c046eab0e3c90b7765857c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f38690805dd641ef81248410cdea033f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3bed3ed17235463782f3dc547ea49637] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=36.1 K 2023-04-22 19:55:01,058 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:55:01,058 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/A is initiating minor compaction (all files) 2023-04-22 19:55:01,058 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/A in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:01,058 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/c151c83b980c4d64a066e9a1247d11a7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/bcfb7bcdae6e4e3d9976ddaf1a81c572, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/8d4b421998d448e6bfa5a9aeb794dcb5] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=36.1 K 2023-04-22 19:55:01,059 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 240880a287c046eab0e3c90b7765857c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682193296853 2023-04-22 19:55:01,059 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c151c83b980c4d64a066e9a1247d11a7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682193296853 2023-04-22 19:55:01,059 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f38690805dd641ef81248410cdea033f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682193298062 2023-04-22 19:55:01,059 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting bcfb7bcdae6e4e3d9976ddaf1a81c572, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682193298062 2023-04-22 19:55:01,059 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 3bed3ed17235463782f3dc547ea49637, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682193300199 2023-04-22 19:55:01,060 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 8d4b421998d448e6bfa5a9aeb794dcb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682193300199 2023-04-22 19:55:01,067 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#B#compaction#269 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:01,067 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#A#compaction#270 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:01,076 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/db3147e05f484266a25b4e53d8177039 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/db3147e05f484266a25b4e53d8177039 2023-04-22 19:55:01,076 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/3cc4bc74b32c43299854e8d0b668e23b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3cc4bc74b32c43299854e8d0b668e23b 2023-04-22 19:55:01,080 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/B of fba73082906c456896fddba31efd5393 into db3147e05f484266a25b4e53d8177039(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:01,080 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:01,080 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/A of fba73082906c456896fddba31efd5393 into 3cc4bc74b32c43299854e8d0b668e23b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:01,080 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/B, priority=13, startTime=1682193301056; duration=0sec 2023-04-22 19:55:01,080 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:01,081 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/A, priority=13, startTime=1682193301056; duration=0sec 2023-04-22 19:55:01,081 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:01,081 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:55:01,081 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:01,081 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:55:01,081 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/C is initiating minor compaction (all files) 2023-04-22 19:55:01,082 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/C in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:01,082 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/b9f35a1b4009405c85cf066ed6126dab, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/00c6e812433c4ba2b9b8a242310e6a6d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/a274331d29574c158a4e67f2730a1d52] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=36.1 K 2023-04-22 19:55:01,082 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b9f35a1b4009405c85cf066ed6126dab, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682193296853 2023-04-22 19:55:01,082 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 00c6e812433c4ba2b9b8a242310e6a6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682193298062 2023-04-22 19:55:01,083 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a274331d29574c158a4e67f2730a1d52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682193300199 2023-04-22 19:55:01,089 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#C#compaction#271 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:01,098 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/eee7d521a36c4abc82957c260c8fb337 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/eee7d521a36c4abc82957c260c8fb337 2023-04-22 19:55:01,102 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fba73082906c456896fddba31efd5393/C of fba73082906c456896fddba31efd5393 into eee7d521a36c4abc82957c260c8fb337(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:01,102 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:01,102 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/C, priority=13, startTime=1682193301056; duration=0sec 2023-04-22 19:55:01,102 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:01,141 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:01,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:01,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:01,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:01,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:01,144 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:01,144 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:01,144 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:01,144 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:01,145 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:01,145 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:01,146 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:01,146 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:01,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:01,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:01,146 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,146 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:01,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:01,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:01,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:01,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:01,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:01,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:01,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:01,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:01,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:01,148 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:01,148 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:01,148 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-22 19:55:01,148 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:55:01,148 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:01,148 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:55:01,148 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:01,148 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:55:01,148 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:01,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:55:01,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:01,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:01,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193361446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:01,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193361549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,561 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/1b84f244d169457db7812bd0d7592fa2 2023-04-22 19:55:01,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:01,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193361752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:01,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59364 deadline: 1682193361772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,774 DEBUG [Thread-1682] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:55:01,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:01,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682193361845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:01,847 DEBUG [Thread-1676] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:55:01,971 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/0f7b5da8b8ee4c2d89b4cb0ada0760c0 2023-04-22 19:55:02,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:02,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193362054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,381 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/94930f8c3bdc4589afea406e62b77f98 2023-04-22 19:55:02,386 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/1b84f244d169457db7812bd0d7592fa2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/1b84f244d169457db7812bd0d7592fa2 2023-04-22 19:55:02,389 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/1b84f244d169457db7812bd0d7592fa2, entries=150, sequenceid=341, filesize=11.9 K 2023-04-22 19:55:02,390 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/0f7b5da8b8ee4c2d89b4cb0ada0760c0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/0f7b5da8b8ee4c2d89b4cb0ada0760c0 2023-04-22 19:55:02,394 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/0f7b5da8b8ee4c2d89b4cb0ada0760c0, entries=150, sequenceid=341, filesize=11.9 K 2023-04-22 19:55:02,395 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/94930f8c3bdc4589afea406e62b77f98 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/94930f8c3bdc4589afea406e62b77f98 2023-04-22 19:55:02,400 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/94930f8c3bdc4589afea406e62b77f98, entries=150, sequenceid=341, filesize=11.9 K 2023-04-22 19:55:02,401 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for fba73082906c456896fddba31efd5393 in 1253ms, sequenceid=341, compaction requested=false 2023-04-22 19:55:02,401 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:02,401 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:02,401 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:02,401 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:02,401 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,401 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:02,401 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:02,402 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:02,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5713f028[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:02,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:02,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:02,406 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:02,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:02,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:02,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:02,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:02,406 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,407 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:02,407 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,407 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:02,407 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:02,407 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:02,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:02,410 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:02,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:02,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:02,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:02,410 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:02,412 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:02,412 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:02,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:02,412 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:02,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:02,413 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,414 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:02,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:02,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:02,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:02,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:02,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:02,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:02,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:02,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:02,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:02,416 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:02,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:02,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:02,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:02,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:02,417 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:02,417 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:02,517 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:02,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:02,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:02,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:02,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:02,520 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:02,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:02,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:02,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:02,521 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:02,521 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:02,522 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:02,522 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:02,522 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,523 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:02,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:02,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:02,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:02,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:02,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:02,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:02,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:02,524 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:02,524 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:02,524 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:55:02,524 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:55:02,525 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:02,525 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:55:02,525 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:02,525 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:55:02,525 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:02,532 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/94207a3887d746beaaf5b14964dfb695 2023-04-22 19:55:02,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:55:02,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:02,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193302587, max timestamp=1682193302596Num uniques 859; 2023-04-22 19:55:02,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193302587, max timestamp=1682193302596Num uniques 859; 2023-04-22 19:55:02,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193302587, max timestamp=1682193302596Num uniques 859; 2023-04-22 19:55:02,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:02,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:02,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:02,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:02,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193362641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:02,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193362744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:02,943 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/c741352c700b43e59e599908da38bb63 2023-04-22 19:55:02,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:02,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193362947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:03,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193363251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,356 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/3ba6a7ec764343159e1d7d4490548148 2023-04-22 19:55:03,362 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/94207a3887d746beaaf5b14964dfb695 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/94207a3887d746beaaf5b14964dfb695 2023-04-22 19:55:03,365 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/94207a3887d746beaaf5b14964dfb695, entries=100, sequenceid=351, filesize=9.5 K 2023-04-22 19:55:03,366 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/c741352c700b43e59e599908da38bb63 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/c741352c700b43e59e599908da38bb63 2023-04-22 19:55:03,370 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/c741352c700b43e59e599908da38bb63, entries=100, sequenceid=351, filesize=9.5 K 2023-04-22 19:55:03,371 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/3ba6a7ec764343159e1d7d4490548148 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3ba6a7ec764343159e1d7d4490548148 2023-04-22 19:55:03,374 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3ba6a7ec764343159e1d7d4490548148, entries=100, sequenceid=351, filesize=9.5 K 2023-04-22 19:55:03,375 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fba73082906c456896fddba31efd5393 in 851ms, sequenceid=351, compaction requested=true 2023-04-22 19:55:03,375 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:03,375 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:03,375 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:03,377 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:03,377 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:03,378 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:03,378 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:03,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:03,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cfb1b3c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:03,379 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:03,379 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:03,381 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:03,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:03,381 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:03,381 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:03,381 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:03,382 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,382 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:03,385 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:03,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:03,387 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:03,388 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:03,388 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:03,388 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:03,388 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:03,389 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:03,389 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:03,389 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:03,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:03,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:03,390 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:03,391 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:03,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:03,396 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:03,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:03,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:03,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:03,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:03,396 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:03,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:03,397 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:03,397 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:03,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:03,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:03,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:03,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:03,497 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:03,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:03,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:03,500 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:03,500 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:03,500 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:03,501 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:03,501 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:03,501 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:03,502 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:03,502 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:03,503 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,503 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:03,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:03,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:03,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:03,504 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:03,504 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:03,504 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:03,504 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:03,504 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:03,504 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:03,504 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:03,504 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-22 19:55:03,505 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:55:03,505 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:03,505 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:55:03,505 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:03,505 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:55:03,505 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:03,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:55:03,760 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:03,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:03,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193363778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:03,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193363881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:03,913 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/d903b2cbb92743888c9aa89518591140 2023-04-22 19:55:04,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:04,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193364085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,324 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/a3e3756af6e743268910744d57a2e637 2023-04-22 19:55:04,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:04,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193364386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:04,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682193364585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,589 DEBUG [Thread-1678] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:55:04,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:04,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59392 deadline: 1682193364677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,680 DEBUG [Thread-1674] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18252 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:55:04,738 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/41ba5f07cbf848c2bdbbc4aa0cf86238 2023-04-22 19:55:04,742 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/d903b2cbb92743888c9aa89518591140 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/d903b2cbb92743888c9aa89518591140 2023-04-22 19:55:04,746 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/d903b2cbb92743888c9aa89518591140, entries=150, sequenceid=388, filesize=11.9 K 2023-04-22 19:55:04,746 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/a3e3756af6e743268910744d57a2e637 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/a3e3756af6e743268910744d57a2e637 2023-04-22 19:55:04,750 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/a3e3756af6e743268910744d57a2e637, entries=150, sequenceid=388, filesize=11.9 K 2023-04-22 19:55:04,751 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/41ba5f07cbf848c2bdbbc4aa0cf86238 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/41ba5f07cbf848c2bdbbc4aa0cf86238 2023-04-22 19:55:04,754 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/41ba5f07cbf848c2bdbbc4aa0cf86238, entries=150, sequenceid=388, filesize=11.9 K 2023-04-22 19:55:04,755 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for fba73082906c456896fddba31efd5393 in 1251ms, sequenceid=388, compaction requested=true 2023-04-22 19:55:04,755 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:04,755 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:04,755 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:04,755 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:04,755 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,755 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:04,755 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:04,756 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:04,756 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:04,757 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:04,757 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:04,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:04,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c6512fb[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:04,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:04,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:04,759 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:04,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:04,760 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,760 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:04,762 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:04,762 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:04,762 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,762 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:04,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:04,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:04,764 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:04,764 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:04,764 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:04,764 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:04,764 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:04,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:04,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:04,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:04,766 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:04,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:04,767 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:04,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:04,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:04,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,768 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,770 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:04,770 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:04,770 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:04,770 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:04,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:04,770 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:04,770 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:04,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:04,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:04,770 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:04,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:04,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:04,771 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:04,771 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:04,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:04,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:04,871 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:04,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:04,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:04,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:04,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:04,874 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:04,874 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:04,874 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:04,874 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:04,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:04,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:04,876 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,876 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:04,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:04,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:04,877 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:04,877 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:04,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:04,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:04,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:04,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:04,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:04,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:04,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:04,878 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:04,878 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:04,878 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:04,878 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:04,878 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:04,878 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:04,878 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:04,878 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:55:04,878 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:55:04,878 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:04,878 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:55:04,878 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:04,878 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:55:04,878 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:04,886 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/b0767c3227b94d488ac7d790360f6677 2023-04-22 19:55:04,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:55:04,892 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:04,897 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/69610fa847ab44f5ae7f93c1b2f5778e 2023-04-22 19:55:04,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193304932, max timestamp=1682193304946Num uniques 859; 2023-04-22 19:55:04,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:04,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193304932, max timestamp=1682193304946Num uniques 859; 2023-04-22 19:55:04,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193304932, max timestamp=1682193304946Num uniques 859; 2023-04-22 19:55:04,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:04,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:05,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:05,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193365002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:05,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193365106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,306 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/d4f4ba7591184722a14c131934deccf4 2023-04-22 19:55:05,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:05,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193365309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,311 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/b0767c3227b94d488ac7d790360f6677 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/b0767c3227b94d488ac7d790360f6677 2023-04-22 19:55:05,315 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/b0767c3227b94d488ac7d790360f6677, entries=150, sequenceid=398, filesize=11.9 K 2023-04-22 19:55:05,315 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/69610fa847ab44f5ae7f93c1b2f5778e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/69610fa847ab44f5ae7f93c1b2f5778e 2023-04-22 19:55:05,319 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/69610fa847ab44f5ae7f93c1b2f5778e, entries=150, sequenceid=398, filesize=11.9 K 2023-04-22 19:55:05,321 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/d4f4ba7591184722a14c131934deccf4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d4f4ba7591184722a14c131934deccf4 2023-04-22 19:55:05,325 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d4f4ba7591184722a14c131934deccf4, entries=150, sequenceid=398, filesize=11.9 K 2023-04-22 19:55:05,326 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fba73082906c456896fddba31efd5393 in 448ms, sequenceid=398, compaction requested=true 2023-04-22 19:55:05,326 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:05,326 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:05,326 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:05,326 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:05,326 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,326 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:05,326 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:05,328 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:05,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:05,329 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:05,329 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:05,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:05,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7abe6c09[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:05,330 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:05,330 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:05,331 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:05,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:05,332 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:05,332 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:05,332 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,332 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:05,332 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:05,333 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,333 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:05,333 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:05,333 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:05,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:05,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:05,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:05,336 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:05,336 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:05,336 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:05,336 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:05,336 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:05,338 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:05,338 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:05,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:05,338 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:05,338 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:05,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:05,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:05,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:05,339 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:05,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:05,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:05,343 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:05,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:05,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:05,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:05,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:05,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:05,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:05,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:05,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:05,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:05,343 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:05,343 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:05,444 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:05,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:05,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:05,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:05,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:05,447 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:05,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:05,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:05,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:05,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:05,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:05,448 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,448 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:05,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:05,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:05,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:05,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:05,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:05,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:05,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:05,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:05,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:05,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:05,450 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:05,450 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:05,450 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:05,450 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:05,450 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:05,450 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:05,450 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. started... 2023-04-22 19:55:05,450 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-22 19:55:05,451 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:55:05,451 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:05,451 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:55:05,451 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:05,451 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:55:05,451 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:05,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:55:05,616 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. as already flushing 2023-04-22 19:55:05,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:05,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193365636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:05,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193365740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:05,862 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/f7c6fa0dac95439296e965f5280c3660 2023-04-22 19:55:05,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:05,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193365944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:06,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59418 deadline: 1682193366249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,274 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/3302b47e50c64dd786ea9dc6b94a8228 2023-04-22 19:55:06,338 DEBUG [Thread-1693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:53381 2023-04-22 19:55:06,338 DEBUG [Thread-1693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:06,340 DEBUG [Thread-1691] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:53381 2023-04-22 19:55:06,340 DEBUG [Thread-1691] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:06,340 DEBUG [Thread-1687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x104dbf4e to 127.0.0.1:53381 2023-04-22 19:55:06,340 DEBUG [Thread-1687] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:06,341 DEBUG [Thread-1685] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a24d617 to 127.0.0.1:53381 2023-04-22 19:55:06,341 DEBUG [Thread-1685] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:06,343 DEBUG [Thread-1689] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x753156b2 to 127.0.0.1:53381 2023-04-22 19:55:06,343 DEBUG [Thread-1689] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:06,685 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/ace55adddf494b419839a5902cae315a 2023-04-22 19:55:06,689 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/f7c6fa0dac95439296e965f5280c3660 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f7c6fa0dac95439296e965f5280c3660 2023-04-22 19:55:06,694 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f7c6fa0dac95439296e965f5280c3660, entries=150, sequenceid=435, filesize=11.9 K 2023-04-22 19:55:06,695 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/3302b47e50c64dd786ea9dc6b94a8228 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3302b47e50c64dd786ea9dc6b94a8228 2023-04-22 19:55:06,698 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3302b47e50c64dd786ea9dc6b94a8228, entries=150, sequenceid=435, filesize=11.9 K 2023-04-22 19:55:06,699 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/ace55adddf494b419839a5902cae315a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ace55adddf494b419839a5902cae315a 2023-04-22 19:55:06,702 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ace55adddf494b419839a5902cae315a, entries=150, sequenceid=435, filesize=11.9 K 2023-04-22 19:55:06,703 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for fba73082906c456896fddba31efd5393 in 1252ms, sequenceid=435, compaction requested=true 2023-04-22 19:55:06,703 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:06,703 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:06,703 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:06,703 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:06,703 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,703 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:06,703 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:06,704 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:06,704 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:06,705 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:06,705 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32e451c6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:06,706 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:06,706 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:06,707 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:06,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:06,707 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:06,707 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:06,707 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:06,707 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,707 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:06,712 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:06,712 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,712 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:06,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:06,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:06,712 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:06,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:06,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:06,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:06,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:06,714 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:06,714 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:06,714 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:06,714 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:06,714 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:06,715 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:06,715 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:06,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:06,715 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:06,716 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,717 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,719 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:06,719 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,719 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:06,719 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:06,719 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:06,719 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:06,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:06,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:06,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:06,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:06,719 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:06,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:06,719 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:06,719 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:06,719 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:06,719 DEBUG [Thread-1684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:06,719 DEBUG [Thread-1684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:06,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:06,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:06,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on fba73082906c456896fddba31efd5393 2023-04-22 19:55:06,756 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 19:55:06,757 DEBUG [Thread-1680] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:53381 2023-04-22 19:55:06,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:55:06,757 DEBUG [Thread-1680] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:06,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:06,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:55:06,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:06,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:55:06,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:06,819 DEBUG [Thread-1684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:06,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:07,163 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/52810e39a74849058f7cae24199aa25f 2023-04-22 19:55:07,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/45b5d958a78c4e9bbc4da15146756b8d 2023-04-22 19:55:07,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/7516cc78a19440f3ba1bd599810d66d0 2023-04-22 19:55:07,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/52810e39a74849058f7cae24199aa25f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/52810e39a74849058f7cae24199aa25f 2023-04-22 19:55:07,988 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/52810e39a74849058f7cae24199aa25f, entries=150, sequenceid=446, filesize=11.9 K 2023-04-22 19:55:07,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/45b5d958a78c4e9bbc4da15146756b8d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/45b5d958a78c4e9bbc4da15146756b8d 2023-04-22 19:55:07,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/45b5d958a78c4e9bbc4da15146756b8d, entries=150, sequenceid=446, filesize=11.9 K 2023-04-22 19:55:07,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/7516cc78a19440f3ba1bd599810d66d0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/7516cc78a19440f3ba1bd599810d66d0 2023-04-22 19:55:07,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/7516cc78a19440f3ba1bd599810d66d0, entries=150, sequenceid=446, filesize=11.9 K 2023-04-22 19:55:07,997 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for fba73082906c456896fddba31efd5393 in 1241ms, sequenceid=446, compaction requested=true 2023-04-22 19:55:07,997 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:07,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:07,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:07,997 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 19:55:07,997 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 19:55:07,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:07,999 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83472 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 19:55:07,999 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83472 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 19:55:07,999 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/A is initiating minor compaction (all files) 2023-04-22 19:55:07,999 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/B is initiating minor compaction (all files) 2023-04-22 19:55:07,999 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/A in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:08,000 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/B in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:08,000 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3cc4bc74b32c43299854e8d0b668e23b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/1b84f244d169457db7812bd0d7592fa2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/94207a3887d746beaaf5b14964dfb695, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/d903b2cbb92743888c9aa89518591140, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/b0767c3227b94d488ac7d790360f6677, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f7c6fa0dac95439296e965f5280c3660, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/52810e39a74849058f7cae24199aa25f] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=81.5 K 2023-04-22 19:55:08,000 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/db3147e05f484266a25b4e53d8177039, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/0f7b5da8b8ee4c2d89b4cb0ada0760c0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/c741352c700b43e59e599908da38bb63, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/a3e3756af6e743268910744d57a2e637, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/69610fa847ab44f5ae7f93c1b2f5778e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3302b47e50c64dd786ea9dc6b94a8228, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/45b5d958a78c4e9bbc4da15146756b8d] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=81.5 K 2023-04-22 19:55:08,000 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 3cc4bc74b32c43299854e8d0b668e23b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682193300199 2023-04-22 19:55:08,000 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting db3147e05f484266a25b4e53d8177039, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682193300199 2023-04-22 19:55:08,000 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 1b84f244d169457db7812bd0d7592fa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1682193300308 2023-04-22 19:55:08,000 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0f7b5da8b8ee4c2d89b4cb0ada0760c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1682193300308 2023-04-22 19:55:08,001 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 94207a3887d746beaaf5b14964dfb695, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1682193301440 2023-04-22 19:55:08,001 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting c741352c700b43e59e599908da38bb63, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1682193301440 2023-04-22 19:55:08,001 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting d903b2cbb92743888c9aa89518591140, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1682193302631 2023-04-22 19:55:08,001 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a3e3756af6e743268910744d57a2e637, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1682193302631 2023-04-22 19:55:08,001 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 69610fa847ab44f5ae7f93c1b2f5778e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1682193303768 2023-04-22 19:55:08,001 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting b0767c3227b94d488ac7d790360f6677, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1682193303768 2023-04-22 19:55:08,002 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 3302b47e50c64dd786ea9dc6b94a8228, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1682193304972 2023-04-22 19:55:08,002 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting f7c6fa0dac95439296e965f5280c3660, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1682193304972 2023-04-22 19:55:08,002 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 45b5d958a78c4e9bbc4da15146756b8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1682193305628 2023-04-22 19:55:08,002 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 52810e39a74849058f7cae24199aa25f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1682193305628 2023-04-22 19:55:08,013 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#B#compaction#290 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:08,013 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#A#compaction#291 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:08,420 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/02936ff0ff87424f8ba49b62accbe11e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/02936ff0ff87424f8ba49b62accbe11e 2023-04-22 19:55:08,420 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/ebdb7f22149d4bda9956ca103fc825b2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/ebdb7f22149d4bda9956ca103fc825b2 2023-04-22 19:55:08,425 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in fba73082906c456896fddba31efd5393/B of fba73082906c456896fddba31efd5393 into ebdb7f22149d4bda9956ca103fc825b2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:08,425 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in fba73082906c456896fddba31efd5393/A of fba73082906c456896fddba31efd5393 into 02936ff0ff87424f8ba49b62accbe11e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:08,425 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:08,425 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:08,425 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/B, priority=9, startTime=1682193307997; duration=0sec 2023-04-22 19:55:08,425 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/A, priority=9, startTime=1682193307997; duration=0sec 2023-04-22 19:55:08,425 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:08,425 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:08,425 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 19:55:08,426 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83472 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 19:55:08,426 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): fba73082906c456896fddba31efd5393/C is initiating minor compaction (all files) 2023-04-22 19:55:08,427 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fba73082906c456896fddba31efd5393/C in TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:08,427 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/eee7d521a36c4abc82957c260c8fb337, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/94930f8c3bdc4589afea406e62b77f98, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3ba6a7ec764343159e1d7d4490548148, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/41ba5f07cbf848c2bdbbc4aa0cf86238, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d4f4ba7591184722a14c131934deccf4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ace55adddf494b419839a5902cae315a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/7516cc78a19440f3ba1bd599810d66d0] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp, totalSize=81.5 K 2023-04-22 19:55:08,427 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting eee7d521a36c4abc82957c260c8fb337, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682193300199 2023-04-22 19:55:08,427 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 94930f8c3bdc4589afea406e62b77f98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1682193300308 2023-04-22 19:55:08,427 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 3ba6a7ec764343159e1d7d4490548148, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1682193301440 2023-04-22 19:55:08,428 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 41ba5f07cbf848c2bdbbc4aa0cf86238, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1682193302631 2023-04-22 19:55:08,428 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting d4f4ba7591184722a14c131934deccf4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1682193303768 2023-04-22 19:55:08,428 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ace55adddf494b419839a5902cae315a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1682193304972 2023-04-22 19:55:08,428 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 7516cc78a19440f3ba1bd599810d66d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1682193305628 2023-04-22 19:55:08,437 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): fba73082906c456896fddba31efd5393#C#compaction#292 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:08,845 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/50a0d490b7754fb59d0f6633db9e5598 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/50a0d490b7754fb59d0f6633db9e5598 2023-04-22 19:55:08,848 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in fba73082906c456896fddba31efd5393/C of fba73082906c456896fddba31efd5393 into 50a0d490b7754fb59d0f6633db9e5598(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:08,848 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:08,848 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393., storeName=fba73082906c456896fddba31efd5393/C, priority=9, startTime=1682193307997; duration=0sec 2023-04-22 19:55:08,848 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:11,858 DEBUG [Thread-1682] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:53381 2023-04-22 19:55:11,859 DEBUG [Thread-1682] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:11,946 DEBUG [Thread-1676] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ed1dde9 to 127.0.0.1:53381 2023-04-22 19:55:11,946 DEBUG [Thread-1676] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:14,659 DEBUG [Thread-1678] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:53381 2023-04-22 19:55:14,659 DEBUG [Thread-1678] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:14,772 DEBUG [Thread-1674] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x112921f2 to 127.0.0.1:53381 2023-04-22 19:55:14,773 DEBUG [Thread-1674] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 9 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 10 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 252 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 42 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 5674 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 5466 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 5581 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 5696 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 5444 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 19:55:14,773 INFO [Listener at localhost/46665] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 19:55:14,773 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x11545f45 to 127.0.0.1:53381 2023-04-22 19:55:14,773 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:14,774 INFO [Listener at localhost/46665] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 19:55:14,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 19:55:14,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:14,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 19:55:14,778 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193314778"}]},"ts":"1682193314778"} 2023-04-22 19:55:14,779 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 19:55:14,780 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 19:55:14,781 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fba73082906c456896fddba31efd5393, UNASSIGN}] 2023-04-22 19:55:14,782 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fba73082906c456896fddba31efd5393, UNASSIGN 2023-04-22 19:55:14,782 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=fba73082906c456896fddba31efd5393, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:14,782 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193314782"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193314782"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193314782"}]},"ts":"1682193314782"} 2023-04-22 19:55:14,783 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:55:14,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 19:55:14,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fba73082906c456896fddba31efd5393 2023-04-22 19:55:14,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fba73082906c456896fddba31efd5393, disabling compactions & flushes 2023-04-22 19:55:14,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:14,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:14,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. after waiting 0 ms 2023-04-22 19:55:14,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:14,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing fba73082906c456896fddba31efd5393 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-22 19:55:14,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=A 2023-04-22 19:55:14,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:14,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=B 2023-04-22 19:55:14,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:14,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fba73082906c456896fddba31efd5393, store=C 2023-04-22 19:55:14,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:15,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 19:55:15,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/3224e2bc4d74416589b24547d2810c86 2023-04-22 19:55:15,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 19:55:15,753 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/6ee307c3a8db4191b791f3d1e8708f97 2023-04-22 19:55:15,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 19:55:16,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/30d209b0c3d04f85a0176cb14fd7e296 2023-04-22 19:55:16,166 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/A/3224e2bc4d74416589b24547d2810c86 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3224e2bc4d74416589b24547d2810c86 2023-04-22 19:55:16,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3224e2bc4d74416589b24547d2810c86, entries=150, sequenceid=456, filesize=11.9 K 2023-04-22 19:55:16,170 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/B/6ee307c3a8db4191b791f3d1e8708f97 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/6ee307c3a8db4191b791f3d1e8708f97 2023-04-22 19:55:16,174 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/6ee307c3a8db4191b791f3d1e8708f97, entries=150, sequenceid=456, filesize=11.9 K 2023-04-22 19:55:16,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/.tmp/C/30d209b0c3d04f85a0176cb14fd7e296 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/30d209b0c3d04f85a0176cb14fd7e296 2023-04-22 19:55:16,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/30d209b0c3d04f85a0176cb14fd7e296, entries=150, sequenceid=456, filesize=11.9 K 2023-04-22 19:55:16,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for fba73082906c456896fddba31efd5393 in 1242ms, sequenceid=456, compaction requested=false 2023-04-22 19:55:16,202 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/fb70f378391f49cabd9a47051567ba82, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/6d451bfa530f453f95d3feac42c71d7d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3c18d4ebcb3848a59d3a11113fca3a3c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/7b161d84514b46e59cb009ea183436f3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/c3b16feea4f44bee9a1e5927ded2c41e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/49b817bc73bf4fd8963df8fb1e0ca41a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/deeefbc621084a54a28e92e272c03eed, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/cef6ca67f07d4303bac97b8e2d737e63, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/2febaeacf5544b8097f7711fd8576a24, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/4187984e09904cee841d51913dcb97c0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f8e925e60bb0481a89dd28dfef31ad9e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/9c657455d4af48539137275563f924e2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/e267ab2fb428459c82bef2ffa3f68127, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/5fb76ae0b7374a3fa937231ea58592e4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/547a0d5ef59a4d1f93d7ca7a541973b6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/c151c83b980c4d64a066e9a1247d11a7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/bcfb7bcdae6e4e3d9976ddaf1a81c572, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3cc4bc74b32c43299854e8d0b668e23b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/8d4b421998d448e6bfa5a9aeb794dcb5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/1b84f244d169457db7812bd0d7592fa2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/94207a3887d746beaaf5b14964dfb695, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/d903b2cbb92743888c9aa89518591140, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/b0767c3227b94d488ac7d790360f6677, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f7c6fa0dac95439296e965f5280c3660, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/52810e39a74849058f7cae24199aa25f] to archive 2023-04-22 19:55:16,203 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:55:16,204 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/fb70f378391f49cabd9a47051567ba82 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/fb70f378391f49cabd9a47051567ba82 2023-04-22 19:55:16,205 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/6d451bfa530f453f95d3feac42c71d7d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/6d451bfa530f453f95d3feac42c71d7d 2023-04-22 19:55:16,206 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3c18d4ebcb3848a59d3a11113fca3a3c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3c18d4ebcb3848a59d3a11113fca3a3c 2023-04-22 19:55:16,207 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/7b161d84514b46e59cb009ea183436f3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/7b161d84514b46e59cb009ea183436f3 2023-04-22 19:55:16,208 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/c3b16feea4f44bee9a1e5927ded2c41e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/c3b16feea4f44bee9a1e5927ded2c41e 2023-04-22 19:55:16,209 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/49b817bc73bf4fd8963df8fb1e0ca41a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/49b817bc73bf4fd8963df8fb1e0ca41a 2023-04-22 19:55:16,210 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/deeefbc621084a54a28e92e272c03eed to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/deeefbc621084a54a28e92e272c03eed 2023-04-22 19:55:16,212 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/cef6ca67f07d4303bac97b8e2d737e63 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/cef6ca67f07d4303bac97b8e2d737e63 2023-04-22 19:55:16,213 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/2febaeacf5544b8097f7711fd8576a24 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/2febaeacf5544b8097f7711fd8576a24 2023-04-22 19:55:16,214 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/4187984e09904cee841d51913dcb97c0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/4187984e09904cee841d51913dcb97c0 2023-04-22 19:55:16,215 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f8e925e60bb0481a89dd28dfef31ad9e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f8e925e60bb0481a89dd28dfef31ad9e 2023-04-22 19:55:16,216 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/9c657455d4af48539137275563f924e2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/9c657455d4af48539137275563f924e2 2023-04-22 19:55:16,217 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/e267ab2fb428459c82bef2ffa3f68127 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/e267ab2fb428459c82bef2ffa3f68127 2023-04-22 19:55:16,218 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/5fb76ae0b7374a3fa937231ea58592e4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/5fb76ae0b7374a3fa937231ea58592e4 2023-04-22 19:55:16,219 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/547a0d5ef59a4d1f93d7ca7a541973b6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/547a0d5ef59a4d1f93d7ca7a541973b6 2023-04-22 19:55:16,220 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/c151c83b980c4d64a066e9a1247d11a7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/c151c83b980c4d64a066e9a1247d11a7 2023-04-22 19:55:16,221 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/bcfb7bcdae6e4e3d9976ddaf1a81c572 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/bcfb7bcdae6e4e3d9976ddaf1a81c572 2023-04-22 19:55:16,221 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3cc4bc74b32c43299854e8d0b668e23b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3cc4bc74b32c43299854e8d0b668e23b 2023-04-22 19:55:16,222 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/8d4b421998d448e6bfa5a9aeb794dcb5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/8d4b421998d448e6bfa5a9aeb794dcb5 2023-04-22 19:55:16,223 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/1b84f244d169457db7812bd0d7592fa2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/1b84f244d169457db7812bd0d7592fa2 2023-04-22 19:55:16,224 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/94207a3887d746beaaf5b14964dfb695 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/94207a3887d746beaaf5b14964dfb695 2023-04-22 19:55:16,225 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/d903b2cbb92743888c9aa89518591140 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/d903b2cbb92743888c9aa89518591140 2023-04-22 19:55:16,225 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/b0767c3227b94d488ac7d790360f6677 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/b0767c3227b94d488ac7d790360f6677 2023-04-22 19:55:16,226 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f7c6fa0dac95439296e965f5280c3660 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/f7c6fa0dac95439296e965f5280c3660 2023-04-22 19:55:16,227 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/52810e39a74849058f7cae24199aa25f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/52810e39a74849058f7cae24199aa25f 2023-04-22 19:55:16,229 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f32ca17f508142aeab1dba6e8bee1302, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/083a25836f3f4a0bbb1545fc183a4693, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/6fbaa414f9bf4a21986e5cdef7512c14, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/b9d5c38b77744c1a8b2b6529a6a6ffaf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/2a303e70ac5746ec99eaae07934e4bc0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/1983138df74141dbb88f6b519d85950c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/cb77417aee184e238be4c4e23fdba735, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/af4ce75135ef4f7cb1f5b9bbf56628b2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/12016cb4239a40c7a40d4a0a68974a1e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3f00db7da25548938a24e3da15ef8294, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/ea28156148764f759a569f035024447c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/cccd46f8b7c941ac83cb4ee084804476, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/609b810bb775455cbc6ef4df55ab3312, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f74b47f9eddc47bb8fec30b77d8c44cb, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/240880a287c046eab0e3c90b7765857c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/08d96db4c43b46b69ff5237fceeb89d4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f38690805dd641ef81248410cdea033f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/db3147e05f484266a25b4e53d8177039, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3bed3ed17235463782f3dc547ea49637, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/0f7b5da8b8ee4c2d89b4cb0ada0760c0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/c741352c700b43e59e599908da38bb63, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/a3e3756af6e743268910744d57a2e637, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/69610fa847ab44f5ae7f93c1b2f5778e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3302b47e50c64dd786ea9dc6b94a8228, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/45b5d958a78c4e9bbc4da15146756b8d] to archive 2023-04-22 19:55:16,230 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:55:16,231 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f32ca17f508142aeab1dba6e8bee1302 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f32ca17f508142aeab1dba6e8bee1302 2023-04-22 19:55:16,232 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/083a25836f3f4a0bbb1545fc183a4693 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/083a25836f3f4a0bbb1545fc183a4693 2023-04-22 19:55:16,233 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/6fbaa414f9bf4a21986e5cdef7512c14 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/6fbaa414f9bf4a21986e5cdef7512c14 2023-04-22 19:55:16,234 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/b9d5c38b77744c1a8b2b6529a6a6ffaf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/b9d5c38b77744c1a8b2b6529a6a6ffaf 2023-04-22 19:55:16,235 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/2a303e70ac5746ec99eaae07934e4bc0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/2a303e70ac5746ec99eaae07934e4bc0 2023-04-22 19:55:16,236 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/1983138df74141dbb88f6b519d85950c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/1983138df74141dbb88f6b519d85950c 2023-04-22 19:55:16,237 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/cb77417aee184e238be4c4e23fdba735 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/cb77417aee184e238be4c4e23fdba735 2023-04-22 19:55:16,237 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/af4ce75135ef4f7cb1f5b9bbf56628b2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/af4ce75135ef4f7cb1f5b9bbf56628b2 2023-04-22 19:55:16,238 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/12016cb4239a40c7a40d4a0a68974a1e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/12016cb4239a40c7a40d4a0a68974a1e 2023-04-22 19:55:16,239 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3f00db7da25548938a24e3da15ef8294 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3f00db7da25548938a24e3da15ef8294 2023-04-22 19:55:16,239 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/ea28156148764f759a569f035024447c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/ea28156148764f759a569f035024447c 2023-04-22 19:55:16,240 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/cccd46f8b7c941ac83cb4ee084804476 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/cccd46f8b7c941ac83cb4ee084804476 2023-04-22 19:55:16,241 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/609b810bb775455cbc6ef4df55ab3312 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/609b810bb775455cbc6ef4df55ab3312 2023-04-22 19:55:16,242 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f74b47f9eddc47bb8fec30b77d8c44cb to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f74b47f9eddc47bb8fec30b77d8c44cb 2023-04-22 19:55:16,242 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/240880a287c046eab0e3c90b7765857c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/240880a287c046eab0e3c90b7765857c 2023-04-22 19:55:16,243 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/08d96db4c43b46b69ff5237fceeb89d4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/08d96db4c43b46b69ff5237fceeb89d4 2023-04-22 19:55:16,244 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f38690805dd641ef81248410cdea033f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/f38690805dd641ef81248410cdea033f 2023-04-22 19:55:16,245 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/db3147e05f484266a25b4e53d8177039 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/db3147e05f484266a25b4e53d8177039 2023-04-22 19:55:16,245 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3bed3ed17235463782f3dc547ea49637 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3bed3ed17235463782f3dc547ea49637 2023-04-22 19:55:16,246 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/0f7b5da8b8ee4c2d89b4cb0ada0760c0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/0f7b5da8b8ee4c2d89b4cb0ada0760c0 2023-04-22 19:55:16,247 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/c741352c700b43e59e599908da38bb63 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/c741352c700b43e59e599908da38bb63 2023-04-22 19:55:16,248 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/a3e3756af6e743268910744d57a2e637 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/a3e3756af6e743268910744d57a2e637 2023-04-22 19:55:16,248 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/69610fa847ab44f5ae7f93c1b2f5778e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/69610fa847ab44f5ae7f93c1b2f5778e 2023-04-22 19:55:16,249 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3302b47e50c64dd786ea9dc6b94a8228 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/3302b47e50c64dd786ea9dc6b94a8228 2023-04-22 19:55:16,250 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/45b5d958a78c4e9bbc4da15146756b8d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/45b5d958a78c4e9bbc4da15146756b8d 2023-04-22 19:55:16,252 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/065ecf64de204867a0ba0e97db7ffebd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/45bd57515df54889b735157b2686764d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d87705f5f79345e9a0a7a9db2a87aec5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/89c42b30759e4453b45db1c3aface021, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/1e0ea5bd6c15456aa314e8219d0e4165, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/fbed117d8d094d6fbea15e7f47ff1f71, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/0b3ad2e0630a4db284f8bc6cdfdc1d5b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d18af2ad37884c0b99a6e3a2c1d0fc06, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ba15cbccd704476cb377893b3eccab66, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/18e2eb0e365243bab33370334da6a5a0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ee44d66b3e424aec9ed5d5256cb2fdd0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/506c6befd43f4b41be5f5fc3c38ac29d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3dc5e61e0cc0481a94b67bb85256e33a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3c1fbbd332ef447482920c117df9fe7d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/b9f35a1b4009405c85cf066ed6126dab, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/b3577bdb50fc4934ad8af629bde1163e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/00c6e812433c4ba2b9b8a242310e6a6d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/eee7d521a36c4abc82957c260c8fb337, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/a274331d29574c158a4e67f2730a1d52, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/94930f8c3bdc4589afea406e62b77f98, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3ba6a7ec764343159e1d7d4490548148, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/41ba5f07cbf848c2bdbbc4aa0cf86238, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d4f4ba7591184722a14c131934deccf4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ace55adddf494b419839a5902cae315a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/7516cc78a19440f3ba1bd599810d66d0] to archive 2023-04-22 19:55:16,253 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:55:16,254 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/065ecf64de204867a0ba0e97db7ffebd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/065ecf64de204867a0ba0e97db7ffebd 2023-04-22 19:55:16,255 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/45bd57515df54889b735157b2686764d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/45bd57515df54889b735157b2686764d 2023-04-22 19:55:16,255 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d87705f5f79345e9a0a7a9db2a87aec5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d87705f5f79345e9a0a7a9db2a87aec5 2023-04-22 19:55:16,256 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/89c42b30759e4453b45db1c3aface021 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/89c42b30759e4453b45db1c3aface021 2023-04-22 19:55:16,257 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/1e0ea5bd6c15456aa314e8219d0e4165 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/1e0ea5bd6c15456aa314e8219d0e4165 2023-04-22 19:55:16,258 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/fbed117d8d094d6fbea15e7f47ff1f71 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/fbed117d8d094d6fbea15e7f47ff1f71 2023-04-22 19:55:16,259 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/0b3ad2e0630a4db284f8bc6cdfdc1d5b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/0b3ad2e0630a4db284f8bc6cdfdc1d5b 2023-04-22 19:55:16,260 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d18af2ad37884c0b99a6e3a2c1d0fc06 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d18af2ad37884c0b99a6e3a2c1d0fc06 2023-04-22 19:55:16,260 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ba15cbccd704476cb377893b3eccab66 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ba15cbccd704476cb377893b3eccab66 2023-04-22 19:55:16,261 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/18e2eb0e365243bab33370334da6a5a0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/18e2eb0e365243bab33370334da6a5a0 2023-04-22 19:55:16,262 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ee44d66b3e424aec9ed5d5256cb2fdd0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ee44d66b3e424aec9ed5d5256cb2fdd0 2023-04-22 19:55:16,263 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/506c6befd43f4b41be5f5fc3c38ac29d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/506c6befd43f4b41be5f5fc3c38ac29d 2023-04-22 19:55:16,264 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3dc5e61e0cc0481a94b67bb85256e33a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3dc5e61e0cc0481a94b67bb85256e33a 2023-04-22 19:55:16,265 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3c1fbbd332ef447482920c117df9fe7d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3c1fbbd332ef447482920c117df9fe7d 2023-04-22 19:55:16,266 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/b9f35a1b4009405c85cf066ed6126dab to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/b9f35a1b4009405c85cf066ed6126dab 2023-04-22 19:55:16,267 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/b3577bdb50fc4934ad8af629bde1163e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/b3577bdb50fc4934ad8af629bde1163e 2023-04-22 19:55:16,268 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/00c6e812433c4ba2b9b8a242310e6a6d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/00c6e812433c4ba2b9b8a242310e6a6d 2023-04-22 19:55:16,269 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/eee7d521a36c4abc82957c260c8fb337 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/eee7d521a36c4abc82957c260c8fb337 2023-04-22 19:55:16,270 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/a274331d29574c158a4e67f2730a1d52 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/a274331d29574c158a4e67f2730a1d52 2023-04-22 19:55:16,271 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/94930f8c3bdc4589afea406e62b77f98 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/94930f8c3bdc4589afea406e62b77f98 2023-04-22 19:55:16,272 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3ba6a7ec764343159e1d7d4490548148 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/3ba6a7ec764343159e1d7d4490548148 2023-04-22 19:55:16,273 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/41ba5f07cbf848c2bdbbc4aa0cf86238 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/41ba5f07cbf848c2bdbbc4aa0cf86238 2023-04-22 19:55:16,274 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d4f4ba7591184722a14c131934deccf4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/d4f4ba7591184722a14c131934deccf4 2023-04-22 19:55:16,275 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ace55adddf494b419839a5902cae315a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/ace55adddf494b419839a5902cae315a 2023-04-22 19:55:16,276 DEBUG [StoreCloser-TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/7516cc78a19440f3ba1bd599810d66d0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/7516cc78a19440f3ba1bd599810d66d0 2023-04-22 19:55:16,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/recovered.edits/459.seqid, newMaxSeqId=459, maxSeqId=1 2023-04-22 19:55:16,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393. 2023-04-22 19:55:16,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fba73082906c456896fddba31efd5393: 2023-04-22 19:55:16,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fba73082906c456896fddba31efd5393 2023-04-22 19:55:16,283 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=fba73082906c456896fddba31efd5393, regionState=CLOSED 2023-04-22 19:55:16,283 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193316283"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193316283"}]},"ts":"1682193316283"} 2023-04-22 19:55:16,285 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-22 19:55:16,285 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure fba73082906c456896fddba31efd5393, server=jenkins-hbase4.apache.org,44791,1682193215766 in 1.5010 sec 2023-04-22 19:55:16,286 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-22 19:55:16,286 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fba73082906c456896fddba31efd5393, UNASSIGN in 1.5050 sec 2023-04-22 19:55:16,287 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193316287"}]},"ts":"1682193316287"} 2023-04-22 19:55:16,287 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 19:55:16,289 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 19:55:16,290 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5150 sec 2023-04-22 19:55:16,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 19:55:16,882 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-22 19:55:16,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 19:55:16,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:16,885 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:16,886 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:16,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-22 19:55:16,891 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393 2023-04-22 19:55:16,893 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/recovered.edits] 2023-04-22 19:55:16,896 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/02936ff0ff87424f8ba49b62accbe11e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/02936ff0ff87424f8ba49b62accbe11e 2023-04-22 19:55:16,897 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3224e2bc4d74416589b24547d2810c86 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/A/3224e2bc4d74416589b24547d2810c86 2023-04-22 19:55:16,899 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/6ee307c3a8db4191b791f3d1e8708f97 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/6ee307c3a8db4191b791f3d1e8708f97 2023-04-22 19:55:16,899 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/ebdb7f22149d4bda9956ca103fc825b2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/B/ebdb7f22149d4bda9956ca103fc825b2 2023-04-22 19:55:16,902 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/30d209b0c3d04f85a0176cb14fd7e296 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/30d209b0c3d04f85a0176cb14fd7e296 2023-04-22 19:55:16,903 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/50a0d490b7754fb59d0f6633db9e5598 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/C/50a0d490b7754fb59d0f6633db9e5598 2023-04-22 19:55:16,905 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/recovered.edits/459.seqid to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393/recovered.edits/459.seqid 2023-04-22 19:55:16,906 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/fba73082906c456896fddba31efd5393 2023-04-22 19:55:16,906 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:55:16,907 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:16,909 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 19:55:16,910 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 19:55:16,911 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:16,911 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 19:55:16,911 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682193316911"}]},"ts":"9223372036854775807"} 2023-04-22 19:55:16,912 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 19:55:16,912 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => fba73082906c456896fddba31efd5393, NAME => 'TestAcidGuarantees,,1682193284168.fba73082906c456896fddba31efd5393.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 19:55:16,912 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 19:55:16,913 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682193316912"}]},"ts":"9223372036854775807"} 2023-04-22 19:55:16,914 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 19:55:16,916 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:16,916 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-22 19:55:16,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-22 19:55:16,987 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-22 19:55:16,998 INFO [Listener at localhost/46665] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 224), OpenFileDescriptor=527 (was 503) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=291 (was 343), ProcessCount=170 (was 170), AvailableMemoryMB=12508 (was 12538) 2023-04-22 19:55:17,009 INFO [Listener at localhost/46665] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=527, MaxFileDescriptor=60000, SystemLoadAverage=291, ProcessCount=170, AvailableMemoryMB=12507 2023-04-22 19:55:17,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 19:55:17,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 19:55:17,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:17,013 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 19:55:17,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-22 19:55:17,014 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 19:55:17,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 19:55:17,015 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:17,015 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852 empty. 2023-04-22 19:55:17,016 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:17,016 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:55:17,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 19:55:17,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 19:55:17,423 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 19:55:17,424 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 75f361e1b049d8d7dacb8de0d0756852, NAME => 'TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp 2023-04-22 19:55:17,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 19:55:17,833 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:55:17,833 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 75f361e1b049d8d7dacb8de0d0756852, disabling compactions & flushes 2023-04-22 19:55:17,833 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:17,833 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:17,833 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. after waiting 0 ms 2023-04-22 19:55:17,833 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:17,833 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:17,833 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:17,835 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 19:55:17,837 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193317836"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193317836"}]},"ts":"1682193317836"} 2023-04-22 19:55:17,838 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 19:55:17,839 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 19:55:17,839 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193317839"}]},"ts":"1682193317839"} 2023-04-22 19:55:17,840 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 19:55:17,842 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=75f361e1b049d8d7dacb8de0d0756852, ASSIGN}] 2023-04-22 19:55:17,844 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=75f361e1b049d8d7dacb8de0d0756852, ASSIGN 2023-04-22 19:55:17,845 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=75f361e1b049d8d7dacb8de0d0756852, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44791,1682193215766; forceNewPlan=false, retain=false 2023-04-22 19:55:17,996 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=75f361e1b049d8d7dacb8de0d0756852, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:17,996 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193317996"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193317996"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193317996"}]},"ts":"1682193317996"} 2023-04-22 19:55:17,998 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:55:18,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 19:55:18,153 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:18,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 75f361e1b049d8d7dacb8de0d0756852, NAME => 'TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:55:18,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:18,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:55:18,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:18,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:18,154 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:18,156 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A 2023-04-22 19:55:18,156 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A 2023-04-22 19:55:18,156 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:55:18,156 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 75f361e1b049d8d7dacb8de0d0756852 columnFamilyName A 2023-04-22 19:55:18,157 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(310): Store=75f361e1b049d8d7dacb8de0d0756852/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:55:18,157 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:18,158 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B 2023-04-22 19:55:18,158 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B 2023-04-22 19:55:18,158 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:55:18,158 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 75f361e1b049d8d7dacb8de0d0756852 columnFamilyName B 2023-04-22 19:55:18,159 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(310): Store=75f361e1b049d8d7dacb8de0d0756852/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:55:18,159 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:18,160 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C 2023-04-22 19:55:18,160 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C 2023-04-22 19:55:18,160 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:55:18,161 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 75f361e1b049d8d7dacb8de0d0756852 columnFamilyName C 2023-04-22 19:55:18,161 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(310): Store=75f361e1b049d8d7dacb8de0d0756852/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:55:18,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:18,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:18,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:18,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 19:55:18,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:18,168 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 19:55:18,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 75f361e1b049d8d7dacb8de0d0756852; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62229139, jitterRate=-0.07271356880664825}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 19:55:18,168 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:18,169 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., pid=37, masterSystemTime=1682193318149 2023-04-22 19:55:18,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:18,171 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:18,171 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=75f361e1b049d8d7dacb8de0d0756852, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:18,171 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193318171"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682193318171"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682193318171"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682193318171"}]},"ts":"1682193318171"} 2023-04-22 19:55:18,174 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-22 19:55:18,174 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 in 174 msec 2023-04-22 19:55:18,176 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-22 19:55:18,176 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=75f361e1b049d8d7dacb8de0d0756852, ASSIGN in 332 msec 2023-04-22 19:55:18,177 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 19:55:18,177 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193318177"}]},"ts":"1682193318177"} 2023-04-22 19:55:18,178 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 19:55:18,180 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 19:55:18,181 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1690 sec 2023-04-22 19:55:19,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 19:55:19,119 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-22 19:55:19,119 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x234f829f to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:19,126 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@279325e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:19,128 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:19,130 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40248, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:19,131 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 19:55:19,132 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45288, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 19:55:19,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 19:55:19,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 19:55:19,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:19,545 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 19:55:19,546 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 19:55:19,546 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 19:55:19,548 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-22 19:55:19,550 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=75f361e1b049d8d7dacb8de0d0756852, REOPEN/MOVE}] 2023-04-22 19:55:19,551 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=75f361e1b049d8d7dacb8de0d0756852, REOPEN/MOVE 2023-04-22 19:55:19,552 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=75f361e1b049d8d7dacb8de0d0756852, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:19,552 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193319552"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193319552"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193319552"}]},"ts":"1682193319552"} 2023-04-22 19:55:19,553 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:55:19,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:19,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 75f361e1b049d8d7dacb8de0d0756852, disabling compactions & flushes 2023-04-22 19:55:19,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:19,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:19,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. after waiting 0 ms 2023-04-22 19:55:19,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:19,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-22 19:55:19,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:19,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:19,710 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 75f361e1b049d8d7dacb8de0d0756852 to self. 2023-04-22 19:55:19,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:19,712 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=75f361e1b049d8d7dacb8de0d0756852, regionState=CLOSED 2023-04-22 19:55:19,712 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193319712"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193319712"}]},"ts":"1682193319712"} 2023-04-22 19:55:19,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-22 19:55:19,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 in 160 msec 2023-04-22 19:55:19,715 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=75f361e1b049d8d7dacb8de0d0756852, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,44791,1682193215766; forceNewPlan=false, retain=true 2023-04-22 19:55:19,865 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=75f361e1b049d8d7dacb8de0d0756852, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:19,865 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193319865"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193319865"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193319865"}]},"ts":"1682193319865"} 2023-04-22 19:55:19,867 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:55:20,021 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 75f361e1b049d8d7dacb8de0d0756852, NAME => 'TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:55:20,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:20,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:55:20,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:20,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:20,023 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:20,024 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A 2023-04-22 19:55:20,024 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A 2023-04-22 19:55:20,025 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:55:20,025 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 75f361e1b049d8d7dacb8de0d0756852 columnFamilyName A 2023-04-22 19:55:20,026 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(310): Store=75f361e1b049d8d7dacb8de0d0756852/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:55:20,027 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:20,027 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B 2023-04-22 19:55:20,027 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B 2023-04-22 19:55:20,028 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:55:20,028 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 75f361e1b049d8d7dacb8de0d0756852 columnFamilyName B 2023-04-22 19:55:20,028 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(310): Store=75f361e1b049d8d7dacb8de0d0756852/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:55:20,028 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:20,029 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C 2023-04-22 19:55:20,029 DEBUG [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C 2023-04-22 19:55:20,029 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:55:20,029 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 75f361e1b049d8d7dacb8de0d0756852 columnFamilyName C 2023-04-22 19:55:20,029 INFO [StoreOpener-75f361e1b049d8d7dacb8de0d0756852-1] regionserver.HStore(310): Store=75f361e1b049d8d7dacb8de0d0756852/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:55:20,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:20,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:20,033 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 19:55:20,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:20,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 75f361e1b049d8d7dacb8de0d0756852; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59001634, jitterRate=-0.12080714106559753}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 19:55:20,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:20,035 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., pid=42, masterSystemTime=1682193320018 2023-04-22 19:55:20,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,036 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=75f361e1b049d8d7dacb8de0d0756852, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,037 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193320036"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682193320036"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682193320036"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682193320036"}]},"ts":"1682193320036"} 2023-04-22 19:55:20,038 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-22 19:55:20,038 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 in 170 msec 2023-04-22 19:55:20,040 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-22 19:55:20,040 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=75f361e1b049d8d7dacb8de0d0756852, REOPEN/MOVE in 488 msec 2023-04-22 19:55:20,041 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-22 19:55:20,041 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 492 msec 2023-04-22 19:55:20,043 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 908 msec 2023-04-22 19:55:20,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-22 19:55:20,045 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x1b77d6d7 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:20,050 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4473f45c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:20,051 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x6434ce29 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:20,055 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41507d16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:20,056 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x0bb27fd4 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:20,061 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b5bd0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:20,061 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x445919fd to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:20,068 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c8d927d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:20,068 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x4d010713 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:20,073 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@419b69bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:20,073 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:20,077 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:20,078 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x06a0df34 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:20,083 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe1db6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:20,083 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:20,090 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:20,091 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x21128196 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:20,096 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4366fcdd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:20,097 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e86d9 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:20,101 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a35a28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:20,109 DEBUG [hconnection-0x38b6acb8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:20,109 DEBUG [hconnection-0x65ec906e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:20,110 DEBUG [hconnection-0x65599ce2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:20,110 DEBUG [hconnection-0x72dc7b3c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:20,111 DEBUG [hconnection-0x18905b53-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:20,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:20,111 DEBUG [hconnection-0xdefc48a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:20,111 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40260, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:20,112 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40286, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:20,113 DEBUG [hconnection-0x757ea819-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:20,113 DEBUG [hconnection-0x36ebdfe4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:20,114 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40320, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:20,116 DEBUG [hconnection-0x17d970bc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:20,118 DEBUG [hconnection-0x4ddb84e8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:20,119 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40278, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:20,120 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40276, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:20,120 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:20,120 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40296, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:20,120 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:20,120 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:20,125 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:20,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:20,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:20,129 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:20,129 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:20,129 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:20,129 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:20,136 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:20,138 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:55:20,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:20,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:20,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:20,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:20,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:20,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:20,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:20,139 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:20,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:20,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:20,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:20,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:20,141 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:20,141 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:20,141 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:20,141 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:20,142 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:20,142 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:20,142 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:20,142 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:20,143 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,144 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,144 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:20,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:20,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36c28e5d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:20,146 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:20,146 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,148 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,148 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:20,148 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:20,148 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:20,148 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,148 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:20,149 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:20,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:20,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:20,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:20,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:20,153 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:20,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:20,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:20,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:20,153 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:20,155 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,155 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:20,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,155 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,155 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,157 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,158 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,158 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:20,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,159 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:20,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:20,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:20,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:20,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:20,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:20,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,159 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:20,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,159 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:20,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:20,165 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304225abf77375d254f58805ac5477941060e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225abf77375d254f58805ac5477941060e 2023-04-22 19:55:20,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682193320152, max timestamp=1682193320163Num uniques 859; 2023-04-22 19:55:20,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-22 19:55:20,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682193320152, max timestamp=1682193320163Num uniques 859; 2023-04-22 19:55:20,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682193320152, max timestamp=1682193320163Num uniques 859; 2023-04-22 19:55:20,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-22 19:55:20,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-22 19:55:20,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193380180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193380185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193380186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193380187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193380187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,259 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:20,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:20,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:20,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:20,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:20,265 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:20,265 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:20,265 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:20,265 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:20,265 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,266 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,271 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,271 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:20,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:20,271 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,271 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:20,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:20,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:20,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:20,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:20,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:20,276 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,276 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:20,276 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:20,276 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:20,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,278 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:20,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72f4b0b4[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:20,279 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:20,279 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,280 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,280 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:20,280 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,280 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:20,280 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:20,281 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:20,287 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,287 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,287 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:20,287 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:20,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:20,290 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:20,290 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:20,290 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:20,290 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:20,290 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:20,292 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,292 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,292 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,294 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193380288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193380288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193380288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193380293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193380293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,298 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:20,298 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:20,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:20,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:20,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:20,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:20,298 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,298 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,298 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,298 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,298 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:20,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:20,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:20,299 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:20,298 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,299 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,299 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,399 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:20,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:20,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:20,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:20,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:20,404 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:20,404 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:20,404 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:20,404 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:20,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,406 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,406 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:20,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:20,406 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,406 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:20,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:20,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:20,407 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:20,408 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,408 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:20,408 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:20,408 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:20,410 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,410 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,410 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:20,410 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:20,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b2354c7[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:20,412 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:20,412 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,414 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,414 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:20,414 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,414 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:20,414 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:20,414 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:20,416 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:20,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:20,418 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:20,418 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:20,418 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:20,418 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:20,418 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:20,420 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,420 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,420 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:20,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,422 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,423 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,425 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,425 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,425 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,425 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,425 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:20,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:20,425 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:20,425 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,425 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:20,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:20,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:20,426 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:20,426 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:20,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:20,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:20,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193380497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193380497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193380497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193380497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193380497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,526 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:20,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:20,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:20,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:20,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:20,534 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:20,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:20,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:20,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:20,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,537 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,538 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,538 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:20,539 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:20,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:20,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:20,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:20,541 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:20,548 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,548 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:20,548 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:20,548 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,551 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b594004[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:20,554 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:20,554 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,555 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,555 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:20,555 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:20,555 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:20,555 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,555 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:20,557 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,557 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:20,557 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:20,557 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:20,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:20,561 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:20,561 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:20,561 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:20,561 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:20,561 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:20,564 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,564 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:20,564 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:20,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,565 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,567 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:20,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,573 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,573 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:20,573 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,573 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:20,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:20,573 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:20,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:20,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:20,573 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:20,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:20,574 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:20,574 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:20,576 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/e6662d791b874985b818ef99d61f35dd 2023-04-22 19:55:20,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/20d3fc783d9e466fa6cabfe6ef3b51af 2023-04-22 19:55:20,604 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/bde4c0f55b5c4cb09d545b787ac167c3 2023-04-22 19:55:20,609 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/e6662d791b874985b818ef99d61f35dd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e6662d791b874985b818ef99d61f35dd 2023-04-22 19:55:20,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e6662d791b874985b818ef99d61f35dd, entries=200, sequenceid=14, filesize=31.9 K 2023-04-22 19:55:20,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/20d3fc783d9e466fa6cabfe6ef3b51af as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/20d3fc783d9e466fa6cabfe6ef3b51af 2023-04-22 19:55:20,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/20d3fc783d9e466fa6cabfe6ef3b51af, entries=150, sequenceid=14, filesize=11.6 K 2023-04-22 19:55:20,620 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/bde4c0f55b5c4cb09d545b787ac167c3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/bde4c0f55b5c4cb09d545b787ac167c3 2023-04-22 19:55:20,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/bde4c0f55b5c4cb09d545b787ac167c3, entries=150, sequenceid=14, filesize=11.6 K 2023-04-22 19:55:20,624 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 75f361e1b049d8d7dacb8de0d0756852 in 486ms, sequenceid=14, compaction requested=false 2023-04-22 19:55:20,624 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-22 19:55:20,626 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:20,676 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:20,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:20,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:20,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:20,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:20,680 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:20,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:20,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:20,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:20,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,681 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,681 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:20,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:20,682 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,682 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:20,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:20,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:20,683 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:20,683 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:20,683 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-22 19:55:20,683 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:20,683 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:20,684 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:20,684 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:20,684 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:20,684 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:20,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:20,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:20,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193380810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193380811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193380812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193380813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193380818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193380920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193380920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193380920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193380920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:20,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:20,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193380921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,103 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422a3dc1ddb709943189b5fc085e89675c8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a3dc1ddb709943189b5fc085e89675c8 2023-04-22 19:55:21,110 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/bbfe820aa13a4cffbe1a2405b9b112eb 2023-04-22 19:55:21,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:21,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193381125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:21,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193381126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:21,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193381127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:21,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193381127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:21,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193381127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:21,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193381430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:21,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193381430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:21,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193381431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:21,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193381434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:21,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193381434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,520 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/2122305e7f6a4746a61ef497721500d7 2023-04-22 19:55:21,531 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/4aa6d108e6654049ac35e700490da289 2023-04-22 19:55:21,535 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/bbfe820aa13a4cffbe1a2405b9b112eb as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/bbfe820aa13a4cffbe1a2405b9b112eb 2023-04-22 19:55:21,541 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/bbfe820aa13a4cffbe1a2405b9b112eb, entries=150, sequenceid=49, filesize=25.1 K 2023-04-22 19:55:21,542 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/2122305e7f6a4746a61ef497721500d7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2122305e7f6a4746a61ef497721500d7 2023-04-22 19:55:21,551 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2122305e7f6a4746a61ef497721500d7, entries=150, sequenceid=49, filesize=11.6 K 2023-04-22 19:55:21,552 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/4aa6d108e6654049ac35e700490da289 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/4aa6d108e6654049ac35e700490da289 2023-04-22 19:55:21,556 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/4aa6d108e6654049ac35e700490da289, entries=150, sequenceid=49, filesize=11.6 K 2023-04-22 19:55:21,557 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 75f361e1b049d8d7dacb8de0d0756852 in 874ms, sequenceid=49, compaction requested=false 2023-04-22 19:55:21,557 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:21,557 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:21,557 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:21,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:21,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,560 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:21,560 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:21,561 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:21,562 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:21,563 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:21,563 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:21,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:21,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49cb8b54[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:21,564 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:21,564 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:21,565 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:21,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:21,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:21,565 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:21,565 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:21,565 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:21,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:21,567 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:21,567 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:21,567 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:21,567 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:21,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:21,570 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:21,570 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:21,570 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:21,570 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:21,570 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:21,573 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:21,573 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:21,573 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:21,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:21,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:21,574 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,577 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,582 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:21,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:21,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:21,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:21,583 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:21,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:21,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:21,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:21,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:21,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:21,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:21,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:21,583 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:21,584 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:21,684 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:21,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:21,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:21,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:21,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:21,688 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:21,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:21,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:21,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:21,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:21,689 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:21,690 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:21,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:21,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:21,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:21,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:21,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:21,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:21,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:21,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:21,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:21,691 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:21,691 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:21,691 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:21,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:21,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:21,692 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:55:21,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:21,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:21,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:21,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:21,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:21,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:21,807 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 19:55:21,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:21,939 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:21,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193321947, max timestamp=1682193321963Num uniques 859; 2023-04-22 19:55:21,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:21,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193321947, max timestamp=1682193321963Num uniques 859; 2023-04-22 19:55:21,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:21,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193321947, max timestamp=1682193321963Num uniques 859; 2023-04-22 19:55:21,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:22,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193381992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193381992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193381993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193382000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193382001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,111 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422a6f28ae6e4ff43818e2d3a0b27acbfc0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a6f28ae6e4ff43818e2d3a0b27acbfc0 2023-04-22 19:55:22,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193382104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193382106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193382106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193382108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193382108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,116 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=59, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/b69250230ba142328895f47031187ff4 2023-04-22 19:55:22,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193382316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193382317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193382317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193382317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193382320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,545 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/4d4a8e0265764a7b8b164e0ea834cf2b 2023-04-22 19:55:22,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193382625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193382625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193382625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193382625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:22,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193382626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,956 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/1a251e930369436b816ca050a2c2a8bd 2023-04-22 19:55:22,961 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/b69250230ba142328895f47031187ff4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/b69250230ba142328895f47031187ff4 2023-04-22 19:55:22,966 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/b69250230ba142328895f47031187ff4, entries=150, sequenceid=59, filesize=25.1 K 2023-04-22 19:55:22,967 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/4d4a8e0265764a7b8b164e0ea834cf2b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/4d4a8e0265764a7b8b164e0ea834cf2b 2023-04-22 19:55:22,973 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/4d4a8e0265764a7b8b164e0ea834cf2b, entries=150, sequenceid=59, filesize=11.6 K 2023-04-22 19:55:22,973 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/1a251e930369436b816ca050a2c2a8bd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/1a251e930369436b816ca050a2c2a8bd 2023-04-22 19:55:22,982 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/1a251e930369436b816ca050a2c2a8bd, entries=150, sequenceid=59, filesize=11.6 K 2023-04-22 19:55:22,983 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 75f361e1b049d8d7dacb8de0d0756852 in 1291ms, sequenceid=59, compaction requested=true 2023-04-22 19:55:22,983 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:22,983 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:22,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:22,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:22,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:22,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:22,985 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:22,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:22,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:22,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:22,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:22,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eedf3c9[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:22,988 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:22,988 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:22,989 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:22,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:22,989 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:22,990 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:22,990 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:22,990 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,990 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:22,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:22,991 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:22,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:22,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:22,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:22,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:22,993 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:22,994 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:22,994 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:22,994 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:22,994 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:22,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:22,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:22,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:22,997 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:22,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:22,997 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:22,998 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:22,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:22,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:23,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:23,001 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:23,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:23,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:23,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:23,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:23,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:23,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:23,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:23,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:23,001 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:23,001 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:23,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:23,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:23,101 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:23,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:23,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:23,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:23,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:23,105 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:23,105 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:23,105 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:23,105 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:23,105 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:23,105 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:23,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:23,107 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:23,107 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,107 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:23,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:23,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:23,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:23,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:23,116 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:23,116 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:23,116 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:23,116 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:23,116 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-22 19:55:23,116 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:23,116 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:23,116 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:23,117 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:23,117 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:23,117 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:23,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:23,138 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:23,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193383140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193383142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193383143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193383147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193383149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193383252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193383252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193383252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193383260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193383262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193383461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193383461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193383462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193383473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193383473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,550 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304222e312cabfee7442e918c97113badb8d8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304222e312cabfee7442e918c97113badb8d8 2023-04-22 19:55:23,576 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=96, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/0dfc773a18df4d30a8fc4b45a42736bd 2023-04-22 19:55:23,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193383768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193383769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193383770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193383785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:23,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193383786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:23,989 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/667d3f4b75724394909fb323978eef40 2023-04-22 19:55:24,011 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/62cb16ea76db448cae1e5cda6a9db99c 2023-04-22 19:55:24,015 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/0dfc773a18df4d30a8fc4b45a42736bd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/0dfc773a18df4d30a8fc4b45a42736bd 2023-04-22 19:55:24,025 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/0dfc773a18df4d30a8fc4b45a42736bd, entries=150, sequenceid=96, filesize=25.1 K 2023-04-22 19:55:24,026 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/667d3f4b75724394909fb323978eef40 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/667d3f4b75724394909fb323978eef40 2023-04-22 19:55:24,036 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/667d3f4b75724394909fb323978eef40, entries=150, sequenceid=96, filesize=11.6 K 2023-04-22 19:55:24,037 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/62cb16ea76db448cae1e5cda6a9db99c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/62cb16ea76db448cae1e5cda6a9db99c 2023-04-22 19:55:24,040 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/62cb16ea76db448cae1e5cda6a9db99c, entries=150, sequenceid=96, filesize=11.6 K 2023-04-22 19:55:24,041 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 75f361e1b049d8d7dacb8de0d0756852 in 925ms, sequenceid=96, compaction requested=true 2023-04-22 19:55:24,041 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:24,041 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:24,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:24,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:24,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:24,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:24,043 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,043 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,043 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,043 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:24,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f8909bf[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:24,044 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:24,044 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,046 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:24,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:24,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:24,047 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,047 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:24,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:24,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:24,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:24,048 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:24,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:24,050 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:24,050 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:24,050 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:24,050 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:24,050 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:24,051 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,051 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,051 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,052 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,053 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,055 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,055 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:24,055 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,055 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:24,055 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,055 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,055 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,055 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,055 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:24,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:24,056 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:24,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:24,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:24,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:24,056 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:24,056 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:24,156 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:24,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:24,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:24,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:24,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:24,160 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:24,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:24,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:24,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:24,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,160 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:24,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:24,162 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,163 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,163 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:24,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:24,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:24,164 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:24,164 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:24,165 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:55:24,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:24,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:24,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:24,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:24,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:24,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:24,177 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042264d7e939ebb748869574edc7e75f33ba to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042264d7e939ebb748869574edc7e75f33ba 2023-04-22 19:55:24,182 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=106, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/8d48bcfb1e714f098b5ffd08dcfbac14 2023-04-22 19:55:24,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:24,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:24,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193324301, max timestamp=1682193324309Num uniques 859; 2023-04-22 19:55:24,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:55:24,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193324301, max timestamp=1682193324309Num uniques 859; 2023-04-22 19:55:24,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193324301, max timestamp=1682193324309Num uniques 859; 2023-04-22 19:55:24,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:55:24,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:55:24,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193384335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193384336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193384348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193384349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193384349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193384450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193384450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193384459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193384465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193384465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,591 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/55546ed725c944bd9dd9e449198aa403 2023-04-22 19:55:24,611 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/6bee8f55a2934851abeaa012cfe13c6b 2023-04-22 19:55:24,615 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/8d48bcfb1e714f098b5ffd08dcfbac14 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/8d48bcfb1e714f098b5ffd08dcfbac14 2023-04-22 19:55:24,618 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/8d48bcfb1e714f098b5ffd08dcfbac14, entries=150, sequenceid=106, filesize=25.1 K 2023-04-22 19:55:24,619 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/55546ed725c944bd9dd9e449198aa403 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/55546ed725c944bd9dd9e449198aa403 2023-04-22 19:55:24,631 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/55546ed725c944bd9dd9e449198aa403, entries=150, sequenceid=106, filesize=11.6 K 2023-04-22 19:55:24,633 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/6bee8f55a2934851abeaa012cfe13c6b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/6bee8f55a2934851abeaa012cfe13c6b 2023-04-22 19:55:24,642 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/6bee8f55a2934851abeaa012cfe13c6b, entries=150, sequenceid=106, filesize=11.6 K 2023-04-22 19:55:24,643 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 75f361e1b049d8d7dacb8de0d0756852 in 478ms, sequenceid=106, compaction requested=true 2023-04-22 19:55:24,643 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:24,643 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:24,643 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:24,644 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:24,644 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,644 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:24,644 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:24,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,646 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,646 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,646 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:24,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef13f2a[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:24,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:24,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,650 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,651 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:24,651 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:24,651 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:24,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,651 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:24,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:24,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:24,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,653 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:24,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:24,657 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:24,657 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:24,657 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:24,657 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:24,657 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:24,660 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,660 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,660 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:24,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,661 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,663 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:24,665 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-22 19:55:24,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193324320, max timestamp=1682193324664Num uniques 1068; 2023-04-22 19:55:24,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:24,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-22 19:55:24,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 19:55:24,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:24,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:24,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:24,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 19:55:24,671 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,672 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,672 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,672 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,672 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,672 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,672 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,672 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:24,672 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:24,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:24,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:24,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:24,672 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:24,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:24,672 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:24,672 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:24,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:24,689 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042231a2e21e71554e068fb1425cfffdd7c3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042231a2e21e71554e068fb1425cfffdd7c3 2023-04-22 19:55:24,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193384693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193384693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193384694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193384697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193384698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,772 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:24,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:24,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:24,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:24,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:24,775 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:24,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:24,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:24,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:24,776 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,776 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:24,777 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:24,777 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,777 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:24,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:24,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:24,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:24,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:24,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:24,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:24,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:24,778 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:24,778 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:24,779 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:24,779 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:24,780 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:24,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:24,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bf58e8b[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:24,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,784 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:24,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:24,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:24,788 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:24,788 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:24,788 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:24,788 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:24,788 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:24,789 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,789 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,789 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:24,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,790 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,791 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:24,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:24,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:24,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,793 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:24,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:24,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:24,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:24,794 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:24,794 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:24,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193384804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193384810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193384810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193384815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:24,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193384815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,894 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:24,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:24,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:24,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:24,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:24,898 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:24,898 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:24,898 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:24,898 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:24,898 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,898 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,900 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:24,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:24,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,901 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:24,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:24,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:24,902 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:24,902 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:24,902 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:24,902 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:24,903 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,903 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,904 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,904 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:24,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:24,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a775f87[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:24,905 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:24,905 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,906 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,906 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:24,907 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:24,907 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:24,907 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,907 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:24,908 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:24,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:24,910 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:24,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:24,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:24,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:24,910 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:24,913 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,913 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:24,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,913 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:24,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:24,913 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,917 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:24,917 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:24,917 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:24,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:24,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:24,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:24,918 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,918 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,918 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:24,918 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:24,918 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:24,918 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:24,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:24,918 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,918 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:24,918 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193385012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,018 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:25,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193385014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193385015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:25,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:25,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:25,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:25,021 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:25,021 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,021 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:25,021 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:25,022 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,022 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,023 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:25,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:25,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:25,028 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,028 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:25,028 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:25,028 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:25,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193385024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193385024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:25,036 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:25,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d846d87[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:25,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:25,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,040 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:25,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:25,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:25,040 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,041 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:25,042 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:25,042 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,042 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,042 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:25,045 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:25,045 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:25,045 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:25,045 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:25,045 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:25,047 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,047 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,047 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,047 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,051 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,053 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:25,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:25,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:25,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,053 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:25,054 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:25,054 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,095 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=145, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/dd7161ce3114408990893c1b38e42951 2023-04-22 19:55:25,154 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:25,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:25,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:25,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:25,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:25,158 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:25,158 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,158 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:25,158 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:25,158 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,158 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,163 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,163 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,163 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,163 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:25,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:25,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:25,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:25,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:25,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:25,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:25,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,167 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:25,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a5b0b72[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:25,169 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:25,169 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,171 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:25,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:25,171 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,171 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:25,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:25,172 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:25,172 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,172 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:25,173 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:25,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:25,175 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:25,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:25,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:25,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:25,175 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:25,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,177 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,178 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:25,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:25,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:25,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,182 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,182 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,182 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:25,182 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,182 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:25,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,282 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:25,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:25,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:25,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:25,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:25,285 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:25,285 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,285 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:25,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:25,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,288 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,289 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,289 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:25,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:25,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:25,290 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,290 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:25,290 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:25,290 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:25,292 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:25,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dbd4764[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:25,293 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:25,293 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,294 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:25,294 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,295 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:25,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,296 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:25,296 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,296 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:25,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:25,298 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:25,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:25,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:25,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:25,298 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:25,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,302 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,303 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,305 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:25,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:25,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,305 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:25,305 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:25,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193385319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193385319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193385321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193385331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193385331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,405 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:25,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:25,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:25,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:25,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:25,409 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:25,409 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,409 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:25,409 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:25,409 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,409 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,410 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,411 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,411 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:25,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:25,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:25,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:25,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:25,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:25,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:25,412 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,412 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:25,412 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:25,412 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:25,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:25,413 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,413 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:25,413 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:25,414 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,414 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,414 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,414 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:25,415 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:25,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3db6bc33[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:25,416 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,417 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,418 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:25,418 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:25,418 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,418 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:25,418 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:25,419 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,419 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,419 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,419 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:25,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:25,421 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:25,421 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:25,421 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:25,421 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:25,421 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:25,423 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,423 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,423 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,423 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,424 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:25,426 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:25,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:25,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,427 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,427 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:25,427 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:25,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,506 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/ebe0f36fdb44461591dda7bd3ac402ba 2023-04-22 19:55:25,527 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:25,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:25,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:25,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:25,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:25,530 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:25,530 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,531 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:25,531 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:25,531 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,531 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,535 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,536 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,536 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:25,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:25,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:25,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:25,537 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,537 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:25,537 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:25,537 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:25,538 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,538 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:25,538 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:25,541 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,541 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,542 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,542 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:25,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13700657[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:25,543 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:25,543 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,544 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,544 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:25,544 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,544 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:25,544 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:25,544 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,546 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:25,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:25,549 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:25,549 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:25,549 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:25,549 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:25,549 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:25,550 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,550 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,550 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,550 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,552 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:25,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:25,554 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,554 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:25,554 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:25,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,555 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,555 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,555 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,655 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:25,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:25,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:25,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:25,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:25,658 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:25,658 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,658 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:25,658 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:25,658 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,658 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,660 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,660 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,660 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,660 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:25,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:25,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:25,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:25,670 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,670 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:25,670 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:25,670 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,671 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:25,671 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:25,671 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,671 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:25,671 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:25,674 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,675 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,675 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,675 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:25,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:25,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11729fd0[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:25,676 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:25,676 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,677 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,677 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:25,677 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:25,677 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,677 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:25,677 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:25,678 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:25,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:25,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:25,684 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:25,684 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:25,684 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:25,685 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:25,685 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:25,686 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,686 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,686 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,687 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,695 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:25,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:25,695 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:25,695 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:25,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,795 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:25,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:25,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:25,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:25,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:25,798 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:25,798 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,799 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:25,799 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:25,799 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,799 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,800 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,800 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,800 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,800 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:25,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:25,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:25,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:25,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:25,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:25,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:25,802 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,802 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:25,802 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:25,802 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:25,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:25,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:25,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:25,804 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,804 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,804 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,804 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46a37cfd[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:25,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:25,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,806 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:25,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:25,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:25,807 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,807 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,809 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:25,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:25,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:25,812 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:25,812 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:25,812 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:25,812 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:25,812 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:25,813 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,813 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,813 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:25,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,814 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,817 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,817 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,817 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:25,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:25,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:25,817 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:25,817 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:25,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:25,817 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:25,817 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,817 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:25,818 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:25,818 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,818 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:25,818 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193385829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193385829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193385831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193385839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:25,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193385842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/b049af2203144df39ea931e236b9391f 2023-04-22 19:55:25,918 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:25,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:25,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:25,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/dd7161ce3114408990893c1b38e42951 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/dd7161ce3114408990893c1b38e42951 2023-04-22 19:55:25,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/dd7161ce3114408990893c1b38e42951, entries=250, sequenceid=145, filesize=38.9 K 2023-04-22 19:55:25,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/ebe0f36fdb44461591dda7bd3ac402ba as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ebe0f36fdb44461591dda7bd3ac402ba 2023-04-22 19:55:25,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:25,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:25,925 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:25,925 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,925 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:25,925 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:25,925 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,925 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,926 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,926 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:25,927 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:25,927 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:25,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ebe0f36fdb44461591dda7bd3ac402ba, entries=150, sequenceid=145, filesize=11.8 K 2023-04-22 19:55:25,929 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/b049af2203144df39ea931e236b9391f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b049af2203144df39ea931e236b9391f 2023-04-22 19:55:25,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:25,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:25,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b049af2203144df39ea931e236b9391f, entries=150, sequenceid=145, filesize=11.8 K 2023-04-22 19:55:25,934 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~322.23 KB/329968, currentSize=67.09 KB/68700 for 75f361e1b049d8d7dacb8de0d0756852 in 1269ms, sequenceid=145, compaction requested=true 2023-04-22 19:55:25,934 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:25,934 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:25,935 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:25,935 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 19:55:25,935 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 19:55:25,935 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:25,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:25,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:25,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:25,943 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:25,943 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:25,943 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71466 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 19:55:25,943 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:25,943 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 175550 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 19:55:25,943 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,943 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:25,943 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/B is initiating minor compaction (all files) 2023-04-22 19:55:25,943 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/B in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,943 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/20d3fc783d9e466fa6cabfe6ef3b51af, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2122305e7f6a4746a61ef497721500d7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/4d4a8e0265764a7b8b164e0ea834cf2b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/667d3f4b75724394909fb323978eef40, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/55546ed725c944bd9dd9e449198aa403, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ebe0f36fdb44461591dda7bd3ac402ba] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=69.8 K 2023-04-22 19:55:25,943 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:25,944 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 20d3fc783d9e466fa6cabfe6ef3b51af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682193320128 2023-04-22 19:55:25,943 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/A is initiating minor compaction (all files) 2023-04-22 19:55:25,945 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/A in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:25,945 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e6662d791b874985b818ef99d61f35dd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/bbfe820aa13a4cffbe1a2405b9b112eb, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/b69250230ba142328895f47031187ff4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/0dfc773a18df4d30a8fc4b45a42736bd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/8d48bcfb1e714f098b5ffd08dcfbac14, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/dd7161ce3114408990893c1b38e42951] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=171.4 K 2023-04-22 19:55:25,946 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 2122305e7f6a4746a61ef497721500d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682193320173 2023-04-22 19:55:25,944 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 19:55:25,946 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting e6662d791b874985b818ef99d61f35dd, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682193320128 2023-04-22 19:55:25,946 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:25,946 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting bbfe820aa13a4cffbe1a2405b9b112eb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682193320173 2023-04-22 19:55:25,946 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:25,946 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 4d4a8e0265764a7b8b164e0ea834cf2b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682193320811 2023-04-22 19:55:25,947 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:25,947 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 667d3f4b75724394909fb323978eef40, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682193321990 2023-04-22 19:55:25,947 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting b69250230ba142328895f47031187ff4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682193320811 2023-04-22 19:55:25,947 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:25,948 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:25,948 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:25,948 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 0dfc773a18df4d30a8fc4b45a42736bd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682193321990 2023-04-22 19:55:25,948 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 55546ed725c944bd9dd9e449198aa403, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682193323138 2023-04-22 19:55:25,948 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 8d48bcfb1e714f098b5ffd08dcfbac14, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682193323138 2023-04-22 19:55:25,949 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ebe0f36fdb44461591dda7bd3ac402ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1682193324347 2023-04-22 19:55:25,949 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting dd7161ce3114408990893c1b38e42951, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1682193324320 2023-04-22 19:55:25,986 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#B#compaction#314 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:26,011 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#A#compaction#315 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:26,410 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/da840760ad024b5a8919672627408cdd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/da840760ad024b5a8919672627408cdd 2023-04-22 19:55:26,415 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/B of 75f361e1b049d8d7dacb8de0d0756852 into da840760ad024b5a8919672627408cdd(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:26,415 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:26,415 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/B, priority=10, startTime=1682193325935; duration=0sec 2023-04-22 19:55:26,415 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:26,415 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 19:55:26,416 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304220399ed306c884f42855f333789a56956 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220399ed306c884f42855f333789a56956 2023-04-22 19:55:26,417 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71466 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 19:55:26,417 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/C is initiating minor compaction (all files) 2023-04-22 19:55:26,417 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/C in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:26,417 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/bde4c0f55b5c4cb09d545b787ac167c3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/4aa6d108e6654049ac35e700490da289, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/1a251e930369436b816ca050a2c2a8bd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/62cb16ea76db448cae1e5cda6a9db99c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/6bee8f55a2934851abeaa012cfe13c6b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b049af2203144df39ea931e236b9391f] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=69.8 K 2023-04-22 19:55:26,418 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting bde4c0f55b5c4cb09d545b787ac167c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682193320128 2023-04-22 19:55:26,418 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 4aa6d108e6654049ac35e700490da289, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682193320173 2023-04-22 19:55:26,418 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1a251e930369436b816ca050a2c2a8bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682193320811 2023-04-22 19:55:26,419 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 62cb16ea76db448cae1e5cda6a9db99c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682193321990 2023-04-22 19:55:26,419 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 6bee8f55a2934851abeaa012cfe13c6b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682193323138 2023-04-22 19:55:26,420 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b049af2203144df39ea931e236b9391f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1682193324347 2023-04-22 19:55:26,422 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/05c1d3ecde964872828d46b6f31c6d47 2023-04-22 19:55:26,444 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#C#compaction#318 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:26,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:26,842 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:26,851 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/011a3638535c466fb9b99de651bc1abb 2023-04-22 19:55:26,856 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/52c4c901d49744f9ba4d088ccf2f9a30 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/52c4c901d49744f9ba4d088ccf2f9a30 2023-04-22 19:55:26,863 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/A of 75f361e1b049d8d7dacb8de0d0756852 into 52c4c901d49744f9ba4d088ccf2f9a30(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:26,863 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:26,863 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/A, priority=10, startTime=1682193325934; duration=0sec 2023-04-22 19:55:26,863 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:26,864 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/f62a306d0bde4c81b0e9a235e20109ba 2023-04-22 19:55:26,866 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/dc7b86a2cdb7437b9638e467a5fa2b21 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/dc7b86a2cdb7437b9638e467a5fa2b21 2023-04-22 19:55:26,872 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/05c1d3ecde964872828d46b6f31c6d47 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05c1d3ecde964872828d46b6f31c6d47 2023-04-22 19:55:26,882 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05c1d3ecde964872828d46b6f31c6d47, entries=150, sequenceid=158, filesize=25.3 K 2023-04-22 19:55:26,883 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/011a3638535c466fb9b99de651bc1abb as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/011a3638535c466fb9b99de651bc1abb 2023-04-22 19:55:26,883 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/C of 75f361e1b049d8d7dacb8de0d0756852 into dc7b86a2cdb7437b9638e467a5fa2b21(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:26,884 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:26,884 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/C, priority=10, startTime=1682193325935; duration=0sec 2023-04-22 19:55:26,884 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:26,890 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/011a3638535c466fb9b99de651bc1abb, entries=150, sequenceid=158, filesize=11.8 K 2023-04-22 19:55:26,890 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/f62a306d0bde4c81b0e9a235e20109ba as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/f62a306d0bde4c81b0e9a235e20109ba 2023-04-22 19:55:26,894 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/f62a306d0bde4c81b0e9a235e20109ba, entries=150, sequenceid=158, filesize=11.8 K 2023-04-22 19:55:26,895 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 75f361e1b049d8d7dacb8de0d0756852 in 951ms, sequenceid=158, compaction requested=false 2023-04-22 19:55:26,895 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:26,895 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:26,898 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:26,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:26,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:26,900 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-22 19:55:26,900 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:26,900 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:26,900 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:26,900 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:26,900 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:26,900 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:26,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e7c073e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:26,902 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:26,902 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:26,903 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:26,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:26,904 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:26,904 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:26,904 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:26,904 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,904 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:26,909 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:26,909 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:26,909 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,909 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:26,911 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:26,911 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:26,911 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:26,911 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:26,911 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:26,915 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:26,915 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:26,915 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:26,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:26,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:26,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:26,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193386915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:26,921 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193386915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:26,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:26,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193386916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:26,922 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:26,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193386917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:26,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193386917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,926 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,926 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:26,926 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:26,926 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:26,926 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:26,926 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:26,926 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:26,926 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:26,926 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:26,926 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:26,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:26,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:26,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:26,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:26,926 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:26,927 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:26,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:26,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193387022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193387023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193387023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,027 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:27,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:27,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193387024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193387024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:27,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:27,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:27,030 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:27,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:27,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:27,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,031 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,032 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,032 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:27,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:27,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:27,033 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,033 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:27,033 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:27,034 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,034 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:27,034 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:27,034 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,034 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:27,034 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:27,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,035 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,035 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,035 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dade218[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:27,037 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:27,037 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,039 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,039 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:27,039 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:27,039 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:27,039 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,039 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:27,040 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:27,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:27,043 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:27,043 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:27,043 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:27,043 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:27,043 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:27,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,045 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,045 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,047 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,049 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,049 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,049 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,049 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,049 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,049 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,049 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,049 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:27,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:27,049 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,049 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:27,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,049 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:27,050 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:27,150 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:27,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:27,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:27,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:27,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:27,154 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:27,154 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,154 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:27,154 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:27,155 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,155 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,158 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,158 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,158 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,158 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:27,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:27,160 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,160 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:27,160 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:27,160 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:27,161 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,162 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,162 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,162 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58a47a62[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:27,164 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:27,164 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,166 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,166 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,166 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:27,166 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:27,166 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:27,166 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:27,168 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,168 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:27,168 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,168 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:27,170 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:27,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:27,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:27,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:27,170 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:27,173 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,173 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,173 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,174 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:27,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:27,178 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,178 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,178 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,178 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,178 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,178 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,179 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:27,179 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:27,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193387227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193387227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193387228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193387229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193387229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,279 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:27,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:27,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:27,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:27,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:27,284 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:27,284 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,284 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:27,284 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:27,285 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,285 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,286 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,287 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,287 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:27,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:27,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:27,292 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,292 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:27,292 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:27,292 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:27,294 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bdb11f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:27,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:27,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,302 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:27,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:27,303 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:27,303 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,303 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:27,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:27,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:27,307 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:27,307 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:27,307 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:27,307 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:27,307 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:27,309 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,309 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,309 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,309 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,310 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,317 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422cea9a8b3213042fa9cace314fb572755 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cea9a8b3213042fa9cace314fb572755 2023-04-22 19:55:27,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,320 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,320 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,320 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,320 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,320 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:27,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:27,320 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:27,320 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:27,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:27,333 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/57360678ac094725bf6639cdd49bae34 2023-04-22 19:55:27,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/6fb23b3fd06040cd8631671ed422f0c3 2023-04-22 19:55:27,420 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:27,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:27,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/7b128e16d326445dadccefc7fe142c61 2023-04-22 19:55:27,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:27,430 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/57360678ac094725bf6639cdd49bae34 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/57360678ac094725bf6639cdd49bae34 2023-04-22 19:55:27,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:27,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:27,432 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:27,432 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,432 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:27,432 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:27,433 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,433 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,433 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/57360678ac094725bf6639cdd49bae34, entries=200, sequenceid=182, filesize=32.1 K 2023-04-22 19:55:27,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/6fb23b3fd06040cd8631671ed422f0c3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/6fb23b3fd06040cd8631671ed422f0c3 2023-04-22 19:55:27,434 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,435 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,435 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,435 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:27,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/6fb23b3fd06040cd8631671ed422f0c3, entries=150, sequenceid=182, filesize=11.8 K 2023-04-22 19:55:27,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/7b128e16d326445dadccefc7fe142c61 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/7b128e16d326445dadccefc7fe142c61 2023-04-22 19:55:27,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:27,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:27,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:27,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:27,451 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:27,451 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:27,452 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,452 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:27,452 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:27,452 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:27,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:27,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:27,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:27,454 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:27,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd61361[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:27,456 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:27,456 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,458 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,458 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:27,458 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:27,458 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:27,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/7b128e16d326445dadccefc7fe142c61, entries=150, sequenceid=182, filesize=11.8 K 2023-04-22 19:55:27,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for 75f361e1b049d8d7dacb8de0d0756852 in 560ms, sequenceid=182, compaction requested=true 2023-04-22 19:55:27,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:27,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:27,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-22 19:55:27,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-22 19:55:27,460 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:55:27,460 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:55:27,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:27,461 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:55:27,461 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/A is initiating minor compaction (all files) 2023-04-22 19:55:27,461 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:55:27,461 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/A in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,461 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/B is initiating minor compaction (all files) 2023-04-22 19:55:27,461 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/52c4c901d49744f9ba4d088ccf2f9a30, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05c1d3ecde964872828d46b6f31c6d47, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/57360678ac094725bf6639cdd49bae34] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=82.9 K 2023-04-22 19:55:27,461 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/B in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,461 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/da840760ad024b5a8919672627408cdd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/011a3638535c466fb9b99de651bc1abb, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/6fb23b3fd06040cd8631671ed422f0c3] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=35.5 K 2023-04-22 19:55:27,462 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 52c4c901d49744f9ba4d088ccf2f9a30, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1682193324347 2023-04-22 19:55:27,462 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting da840760ad024b5a8919672627408cdd, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1682193324347 2023-04-22 19:55:27,462 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 05c1d3ecde964872828d46b6f31c6d47, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682193324691 2023-04-22 19:55:27,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,462 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 011a3638535c466fb9b99de651bc1abb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682193324691 2023-04-22 19:55:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,462 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 57360678ac094725bf6639cdd49bae34, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682193326865 2023-04-22 19:55:27,463 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 6fb23b3fd06040cd8631671ed422f0c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682193326865 2023-04-22 19:55:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,463 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:27,463 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,463 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,473 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#B#compaction#323 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:27,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:27,476 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:27,476 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:27,476 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:27,476 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:27,476 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:27,478 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,478 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,478 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,485 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,490 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,491 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:27,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:27,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,492 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,492 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,492 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:27,492 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:27,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:27,497 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#A#compaction#324 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:27,521 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/9d85a8315d53481cb117a6f9408c29e8 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/9d85a8315d53481cb117a6f9408c29e8 2023-04-22 19:55:27,525 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/B of 75f361e1b049d8d7dacb8de0d0756852 into 9d85a8315d53481cb117a6f9408c29e8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:27,526 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:27,526 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/B, priority=13, startTime=1682193327460; duration=0sec 2023-04-22 19:55:27,526 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:27,526 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:55:27,527 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:55:27,527 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/C is initiating minor compaction (all files) 2023-04-22 19:55:27,527 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/C in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,527 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/dc7b86a2cdb7437b9638e467a5fa2b21, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/f62a306d0bde4c81b0e9a235e20109ba, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/7b128e16d326445dadccefc7fe142c61] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=35.5 K 2023-04-22 19:55:27,528 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting dc7b86a2cdb7437b9638e467a5fa2b21, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1682193324347 2023-04-22 19:55:27,528 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting f62a306d0bde4c81b0e9a235e20109ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682193324691 2023-04-22 19:55:27,529 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 7b128e16d326445dadccefc7fe142c61, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682193326865 2023-04-22 19:55:27,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:27,541 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 19:55:27,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:27,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:27,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:27,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:27,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:27,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:27,557 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#C#compaction#325 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:27,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,578 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/4484a08068e944caab5569bfb3f973dd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/4484a08068e944caab5569bfb3f973dd 2023-04-22 19:55:27,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193387564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193387564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193387565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,583 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/C of 75f361e1b049d8d7dacb8de0d0756852 into 4484a08068e944caab5569bfb3f973dd(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:27,583 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:27,583 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/C, priority=13, startTime=1682193327460; duration=0sec 2023-04-22 19:55:27,583 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:27,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193387576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193387578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,592 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:27,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:27,595 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042208cae03aaa32496f9cee9f0e114620de to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042208cae03aaa32496f9cee9f0e114620de 2023-04-22 19:55:27,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:27,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:27,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:27,604 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:27,604 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,604 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:27,604 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:27,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,607 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,607 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,608 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,608 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:27,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:27,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:27,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:27,612 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:27,612 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,612 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:27,620 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:27,620 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:27,620 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,620 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=196, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/237a506d606c472284e69e455daaf96f 2023-04-22 19:55:27,620 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:27,620 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:27,621 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,621 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:27,621 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:27,621 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,621 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:27,621 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:27,624 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,624 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,624 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,624 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19599921[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:27,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:27,627 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,628 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,628 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:27,628 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:27,628 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:27,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:27,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,629 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,629 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:27,630 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:27,630 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:27,630 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:27,630 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:27,630 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:27,630 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:27,630 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:27,633 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,633 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,633 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,634 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,635 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/debf547051e9449a9a5fd9803eda09a9 2023-04-22 19:55:27,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,638 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:27,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:27,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,638 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:27,638 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:27,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193387679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193387679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193387680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193387686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193387690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,739 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:27,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:27,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:27,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:27,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:27,742 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:27,742 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,742 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:27,742 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:27,743 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,743 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,744 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,744 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,744 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:27,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:27,745 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:27,745 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,745 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:27,746 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,746 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:27,746 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:27,746 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:27,747 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,747 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,747 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,747 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:27,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d59702[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:27,755 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:27,755 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,756 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,756 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:27,756 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:27,756 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,756 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:27,756 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:27,758 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,758 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:27,759 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,759 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:27,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:27,762 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:27,762 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:27,762 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:27,762 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:27,762 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:27,764 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,765 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,765 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,765 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,766 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:27,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:27,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,771 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,772 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:27,772 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:27,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,872 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:27,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:27,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:27,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:27,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:27,875 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:27,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:27,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:27,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,876 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,878 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,878 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,878 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,878 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:27,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:27,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:27,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:27,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:27,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:27,881 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,881 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,882 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,882 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:27,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a88e70e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:27,884 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:27,884 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193387881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193387881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193387882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,886 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,887 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:27,887 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:27,887 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:27,887 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,887 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:27,888 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,888 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:27,888 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,888 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:27,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:27,890 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:27,890 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:27,890 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:27,890 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:27,890 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:27,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,894 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193387891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:27,894 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,894 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:27,896 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,899 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:27,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:27,901 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:27,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:27,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:27,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:27,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:27,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:27,901 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:27,901 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:27,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:27,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:27,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193387899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,002 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:28,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:28,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:28,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:28,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:28,006 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:28,006 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:28,006 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:28,006 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:28,006 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:28,006 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:28,008 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:28,008 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:28,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:28,009 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,009 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:28,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:28,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:28,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:28,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:28,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:28,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:28,016 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:28,016 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:28,016 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:28,016 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:28,017 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:28,018 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:28,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:28,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:28,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:28,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:28,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:28,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:28,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:28,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:28,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:28,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11deb025[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:28,021 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:28,021 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:28,023 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:28,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:28,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:28,023 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:28,023 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:28,023 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:28,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:28,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:28,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:28,024 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:28,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:28,025 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:28,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:28,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:28,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:28,025 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:28,027 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:28,027 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:28,027 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:28,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:28,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:28,028 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:28,031 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,033 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:28,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:28,033 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:28,033 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:28,033 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:28,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:28,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:28,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:28,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:28,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:28,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:28,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:28,034 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:28,034 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:28,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:28,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/ec1a945b38ea4184bd299672b919875a 2023-04-22 19:55:28,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/237a506d606c472284e69e455daaf96f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/237a506d606c472284e69e455daaf96f 2023-04-22 19:55:28,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/237a506d606c472284e69e455daaf96f, entries=200, sequenceid=196, filesize=32.1 K 2023-04-22 19:55:28,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/debf547051e9449a9a5fd9803eda09a9 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/debf547051e9449a9a5fd9803eda09a9 2023-04-22 19:55:28,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/debf547051e9449a9a5fd9803eda09a9, entries=150, sequenceid=196, filesize=11.8 K 2023-04-22 19:55:28,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/ec1a945b38ea4184bd299672b919875a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/ec1a945b38ea4184bd299672b919875a 2023-04-22 19:55:28,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/ec1a945b38ea4184bd299672b919875a, entries=150, sequenceid=196, filesize=11.8 K 2023-04-22 19:55:28,070 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 75f361e1b049d8d7dacb8de0d0756852 in 529ms, sequenceid=196, compaction requested=false 2023-04-22 19:55:28,070 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:28,134 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:28,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:28,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:28,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:28,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:28,138 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:28,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:28,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:28,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:28,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:28,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:28,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,140 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:28,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:28,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:28,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,140 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:28,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:28,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:28,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:28,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:28,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:28,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:28,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:28,141 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:28,141 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:28,141 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:28,142 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:28,142 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:28,142 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:28,142 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:28,142 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 19:55:28,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:28,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:28,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:28,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:28,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:28,143 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:28,156 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422330ae0975c0a44949cf3cd32f31e3f17 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422330ae0975c0a44949cf3cd32f31e3f17 2023-04-22 19:55:28,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:28,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:28,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193388205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193388206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193388207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193388213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193388215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193388316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193388320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193388320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193388325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,361 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/267db742051340969cb9e3eaaf8f59db as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/267db742051340969cb9e3eaaf8f59db 2023-04-22 19:55:28,375 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/A of 75f361e1b049d8d7dacb8de0d0756852 into 267db742051340969cb9e3eaaf8f59db(size=25.6 K), total size for store is 57.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:28,375 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:28,375 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/A, priority=13, startTime=1682193327460; duration=0sec 2023-04-22 19:55:28,375 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:28,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193388520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193388524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193388526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193388529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,561 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=216, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/ddab1a4f68fe45f0b2ffba577c589614 2023-04-22 19:55:28,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193388724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193388828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193388829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193388832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193388832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:28,972 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/2443fed9caf642a8879d0911d84636c7 2023-04-22 19:55:29,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:29,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193389334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:29,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193389335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:29,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193389337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:29,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193389338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,382 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/c17f7a9a6666468b843b36c39efbccfe 2023-04-22 19:55:29,386 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/ddab1a4f68fe45f0b2ffba577c589614 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/ddab1a4f68fe45f0b2ffba577c589614 2023-04-22 19:55:29,390 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/ddab1a4f68fe45f0b2ffba577c589614, entries=150, sequenceid=216, filesize=25.3 K 2023-04-22 19:55:29,390 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/2443fed9caf642a8879d0911d84636c7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2443fed9caf642a8879d0911d84636c7 2023-04-22 19:55:29,394 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2443fed9caf642a8879d0911d84636c7, entries=150, sequenceid=216, filesize=11.8 K 2023-04-22 19:55:29,395 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/c17f7a9a6666468b843b36c39efbccfe as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c17f7a9a6666468b843b36c39efbccfe 2023-04-22 19:55:29,398 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c17f7a9a6666468b843b36c39efbccfe, entries=150, sequenceid=216, filesize=11.8 K 2023-04-22 19:55:29,399 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 75f361e1b049d8d7dacb8de0d0756852 in 1256ms, sequenceid=216, compaction requested=true 2023-04-22 19:55:29,399 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:29,399 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:29,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:29,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:29,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:29,399 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:29,402 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:29,402 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:29,403 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:29,403 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:29,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:29,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@441bb3ae[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:29,404 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:29,404 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:29,405 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:29,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:29,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:29,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:29,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:29,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:29,407 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,407 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:29,408 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:29,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:29,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:29,410 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:29,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:29,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:29,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:29,410 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:29,413 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:29,413 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:29,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:29,415 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:29,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:29,415 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:29,417 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,419 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,419 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:29,419 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:29,419 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:29,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:29,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:29,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:29,419 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,419 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:29,420 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:29,420 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:29,420 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:29,419 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:29,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:29,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:29,420 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:29,420 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:29,520 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:29,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:29,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:29,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:29,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:29,523 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:29,523 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:29,524 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:29,524 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:29,524 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:29,524 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:29,528 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,528 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:29,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:29,528 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,528 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:29,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:29,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:29,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:29,529 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:29,529 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:29,529 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:29,529 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:29,529 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:29,529 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:29,529 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:29,530 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 19:55:29,530 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:29,530 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:29,530 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:29,530 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:29,530 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:29,530 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:29,546 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042241b2fed999d74e44be2a8f824f80f865 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042241b2fed999d74e44be2a8f824f80f865 2023-04-22 19:55:29,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:29,732 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:29,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:29,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193389835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:29,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193389943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:29,953 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=230, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/05e88652f92f4b6fb1cb5087390d6510 2023-04-22 19:55:29,983 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/3cbdaadf1a674972961f2dc58787e63a 2023-04-22 19:55:29,996 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/2ce019ff9ac14a0686665973ae844a4d 2023-04-22 19:55:30,001 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/05e88652f92f4b6fb1cb5087390d6510 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05e88652f92f4b6fb1cb5087390d6510 2023-04-22 19:55:30,004 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05e88652f92f4b6fb1cb5087390d6510, entries=150, sequenceid=230, filesize=25.3 K 2023-04-22 19:55:30,005 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/3cbdaadf1a674972961f2dc58787e63a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/3cbdaadf1a674972961f2dc58787e63a 2023-04-22 19:55:30,008 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/3cbdaadf1a674972961f2dc58787e63a, entries=150, sequenceid=230, filesize=11.8 K 2023-04-22 19:55:30,009 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/2ce019ff9ac14a0686665973ae844a4d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/2ce019ff9ac14a0686665973ae844a4d 2023-04-22 19:55:30,012 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/2ce019ff9ac14a0686665973ae844a4d, entries=150, sequenceid=230, filesize=11.8 K 2023-04-22 19:55:30,013 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 75f361e1b049d8d7dacb8de0d0756852 in 483ms, sequenceid=230, compaction requested=true 2023-04-22 19:55:30,013 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:30,013 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:30,016 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ab851a8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:30,017 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:30,017 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:30,018 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:30,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:30,019 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,020 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:30,020 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:30,022 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:30,022 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:30,022 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:30,022 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:30,022 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:30,023 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:30,023 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:30,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:30,023 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:30,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:30,024 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:30,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,030 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,030 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:30,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:30,031 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:30,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:30,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:30,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:30,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:30,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:30,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:30,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:30,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:30,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:30,031 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:30,031 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:30,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:30,131 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:30,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:30,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:30,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:30,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:30,136 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:30,136 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:30,136 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:30,136 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:30,136 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:30,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:30,138 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:30,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:30,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:30,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:30,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:30,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:30,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:30,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:30,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:30,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:30,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:30,139 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:30,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:30,140 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 19:55:30,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:30,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:30,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:30,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:30,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:30,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:30,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:30,150 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:30,159 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422095bf7754b1f408b81f138e32aa63e7c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422095bf7754b1f408b81f138e32aa63e7c 2023-04-22 19:55:30,169 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=249, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/20b425791bcd47949501e1a1ad1b32fa 2023-04-22 19:55:30,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193390234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:30,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40310 deadline: 1682193390341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:30,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193390344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:30,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193390347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:30,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193390349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:30,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193390350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:30,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193390549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,580 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/1bb1fb10e05743d79dd0e71fb1d2b611 2023-04-22 19:55:30,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:30,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193390856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:30,995 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/77cc462ce6c6472f9249120a154246d2 2023-04-22 19:55:31,000 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/20b425791bcd47949501e1a1ad1b32fa as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/20b425791bcd47949501e1a1ad1b32fa 2023-04-22 19:55:31,003 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/20b425791bcd47949501e1a1ad1b32fa, entries=150, sequenceid=249, filesize=25.3 K 2023-04-22 19:55:31,005 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/1bb1fb10e05743d79dd0e71fb1d2b611 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1bb1fb10e05743d79dd0e71fb1d2b611 2023-04-22 19:55:31,009 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1bb1fb10e05743d79dd0e71fb1d2b611, entries=150, sequenceid=249, filesize=11.8 K 2023-04-22 19:55:31,010 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/77cc462ce6c6472f9249120a154246d2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/77cc462ce6c6472f9249120a154246d2 2023-04-22 19:55:31,013 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/77cc462ce6c6472f9249120a154246d2, entries=150, sequenceid=249, filesize=11.8 K 2023-04-22 19:55:31,014 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 75f361e1b049d8d7dacb8de0d0756852 in 874ms, sequenceid=249, compaction requested=true 2023-04-22 19:55:31,014 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:31,014 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:31,014 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:31,014 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:31,014 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,014 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:31,014 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:31,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,016 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,016 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:31,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:31,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47456766[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:31,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:31,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,019 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:31,020 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:31,020 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,020 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:31,020 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:31,022 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:31,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:31,025 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:31,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:31,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:31,025 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:31,025 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:31,027 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,027 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:31,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,027 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:31,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:31,028 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:31,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:31,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:31,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:31,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:31,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:31,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,032 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:31,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:31,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:31,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,032 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:31,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:31,032 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:31,132 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:31,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:31,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:31,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:31,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:31,136 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:31,136 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:31,136 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:31,136 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:31,136 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,136 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:31,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:31,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:31,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:31,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:31,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:31,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:31,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:31,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:31,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:31,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:31,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:31,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:31,139 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:31,139 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:31,139 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 19:55:31,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:31,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:31,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:31,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:31,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:31,140 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:31,164 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422c3ebef4088ba42c2b0e7d6a38ace6c92 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c3ebef4088ba42c2b0e7d6a38ace6c92 2023-04-22 19:55:31,169 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=262, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/a19e6bc5e79b4fab9ca0727633222317 2023-04-22 19:55:31,208 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/ac4b7d8414324bcca5bffbc323f1beb8 2023-04-22 19:55:31,234 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/cd69669d5e184596882d968f456d46c0 2023-04-22 19:55:31,239 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/a19e6bc5e79b4fab9ca0727633222317 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/a19e6bc5e79b4fab9ca0727633222317 2023-04-22 19:55:31,245 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/a19e6bc5e79b4fab9ca0727633222317, entries=150, sequenceid=262, filesize=25.4 K 2023-04-22 19:55:31,246 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/ac4b7d8414324bcca5bffbc323f1beb8 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ac4b7d8414324bcca5bffbc323f1beb8 2023-04-22 19:55:31,255 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ac4b7d8414324bcca5bffbc323f1beb8, entries=150, sequenceid=262, filesize=11.9 K 2023-04-22 19:55:31,257 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/cd69669d5e184596882d968f456d46c0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/cd69669d5e184596882d968f456d46c0 2023-04-22 19:55:31,274 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/cd69669d5e184596882d968f456d46c0, entries=150, sequenceid=262, filesize=11.9 K 2023-04-22 19:55:31,275 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 75f361e1b049d8d7dacb8de0d0756852 in 136ms, sequenceid=262, compaction requested=true 2023-04-22 19:55:31,275 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:31,275 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:31,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:31,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:31,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,281 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:31,281 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:31,283 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:31,284 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,285 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:31,285 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:31,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:31,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23feb833[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:31,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:31,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:31,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:31,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:31,289 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,290 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:31,290 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:31,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:31,291 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:31,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:31,294 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:31,294 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:31,294 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:31,294 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:31,294 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:31,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:31,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:31,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:31,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:31,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:31,301 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:31,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:31,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:31,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:31,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,302 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:31,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:31,302 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:31,302 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:31,302 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:31,302 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:31,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:31,402 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:31,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:31,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:31,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:31,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:31,433 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:31,433 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:31,433 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:31,433 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:31,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:31,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,437 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:31,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:31,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:31,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:31,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:31,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:31,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:31,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:31,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:31,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:31,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:31,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:31,452 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:31,463 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:31,463 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:31,463 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:31,463 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:31,463 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-22 19:55:31,463 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:31,463 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:31,463 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:31,464 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:31,464 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:31,464 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:31,510 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422f387525c6b38439880fc3c6096e5b5fa to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f387525c6b38439880fc3c6096e5b5fa 2023-04-22 19:55:31,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:31,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:31,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193331669, max timestamp=1682193331721Num uniques 859; 2023-04-22 19:55:31,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:31,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193331669, max timestamp=1682193331721Num uniques 859; 2023-04-22 19:55:31,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:31,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193331669, max timestamp=1682193331721Num uniques 859; 2023-04-22 19:55:31,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:31,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193331862, max timestamp=1682193331912Num uniques 1018; 2023-04-22 19:55:31,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 19:55:31,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193331862, max timestamp=1682193331912Num uniques 1018; 2023-04-22 19:55:31,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 19:55:31,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193331862, max timestamp=1682193331912Num uniques 1018; 2023-04-22 19:55:31,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 19:55:31,957 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=266, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/f993c41490df43cbab703638586b72c9 2023-04-22 19:55:31,971 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/ee6a610c28444e82b4cc2c8f90841600 2023-04-22 19:55:31,988 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/9e93f186a42c4ba9853db8db4367afd4 2023-04-22 19:55:31,993 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/f993c41490df43cbab703638586b72c9 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/f993c41490df43cbab703638586b72c9 2023-04-22 19:55:32,001 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/f993c41490df43cbab703638586b72c9, entries=50, sequenceid=266, filesize=11.7 K 2023-04-22 19:55:32,002 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/ee6a610c28444e82b4cc2c8f90841600 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ee6a610c28444e82b4cc2c8f90841600 2023-04-22 19:55:32,005 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ee6a610c28444e82b4cc2c8f90841600, entries=50, sequenceid=266, filesize=7.1 K 2023-04-22 19:55:32,006 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/9e93f186a42c4ba9853db8db4367afd4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9e93f186a42c4ba9853db8db4367afd4 2023-04-22 19:55:32,010 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9e93f186a42c4ba9853db8db4367afd4, entries=50, sequenceid=266, filesize=7.1 K 2023-04-22 19:55:32,011 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=73.80 KB/75570 for 75f361e1b049d8d7dacb8de0d0756852 in 547ms, sequenceid=266, compaction requested=true 2023-04-22 19:55:32,011 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:32,011 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:32,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,011 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=80.51 KB heapSize=224.77 KB 2023-04-22 19:55:32,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:32,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:32,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:32,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:32,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:32,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:32,012 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:32,018 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,018 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,019 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:32,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ae1f921[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:32,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:32,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,024 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:32,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:32,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:32,026 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,027 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:32,027 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,027 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:32,027 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:32,027 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:32,033 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:32,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:32,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:32,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:32,033 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:32,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:32,035 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042243bf331174714628a497a1e23d3d1391 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042243bf331174714628a497a1e23d3d1391 2023-04-22 19:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:32,036 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:32,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,041 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:32,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:32,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:32,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:32,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:32,041 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:32,042 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:32,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,042 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,042 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,042 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:32,042 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,047 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=312, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/1b07f80edceb4555a6a24a34ebbd4270 2023-04-22 19:55:32,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/56adcae4fba0413d9bd334d865f00407 2023-04-22 19:55:32,142 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:32,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:32,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:32,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:32,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:32,147 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:32,147 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:32,147 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:32,147 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:32,148 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,148 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,149 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:32,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:32,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,149 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:32,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:32,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:32,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:32,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:32,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:32,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:32,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:32,151 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:32,151 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:32,151 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:32,151 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:32,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:32,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:32,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:32,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:32,159 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,159 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,159 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,159 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:32,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@729ccac4[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:32,161 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:32,161 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,162 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:32,162 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,162 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:32,163 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:32,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:32,166 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:32,166 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:32,166 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:32,166 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:32,166 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:32,169 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,169 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,169 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:32,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:32,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,171 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,174 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,174 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:32,174 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:32,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:32,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:32,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:32,174 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:32,174 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,174 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:32,175 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:32,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:32,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:32,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:32,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193392191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,275 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:32,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:32,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:32,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:32,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:32,279 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:32,279 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:32,279 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:32,279 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:32,279 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,280 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,281 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:32,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:32,281 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,281 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:32,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:32,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:32,282 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:32,282 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:32,282 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:32,283 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:32,283 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:32,283 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:32,283 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:32,285 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,285 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,285 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,285 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e0efda6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:32,287 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:32,287 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,289 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,289 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:32,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:32,291 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:32,291 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:32,291 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:32,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:32,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:32,295 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:32,295 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:32,295 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:32,295 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:32,295 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:32,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,296 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,297 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,300 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:32,300 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:32,300 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:32,301 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:32,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:32,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:32,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:32,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,301 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,301 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:32,301 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:32,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:32,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:32,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193392301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:32,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193392355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,368 DEBUG [Thread-2246] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:55:32,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:32,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193392363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:32,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193392363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:32,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193392364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,374 DEBUG [Thread-2240] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:55:32,381 DEBUG [Thread-2244] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:55:32,383 DEBUG [Thread-2248] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:55:32,401 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:32,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:32,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:32,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:32,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:32,405 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:32,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:32,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:32,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:32,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,407 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,407 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:32,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:32,408 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,408 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:32,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:32,417 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:32,417 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:32,417 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:32,417 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:32,419 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,419 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,419 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,419 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:32,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e7e174[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:32,421 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:32,421 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,422 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,422 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:32,422 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:32,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:32,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:32,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,426 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:32,427 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:32,427 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:32,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:32,429 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:32,429 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:32,429 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:32,429 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:32,429 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:32,430 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:32,431 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,431 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:32,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:32,432 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,433 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,435 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,435 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,435 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,435 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,435 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:32,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:32,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:32,435 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:32,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:32,435 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,436 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:32,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:32,436 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:32,436 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:32,436 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:32,436 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:32,503 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/b9f1b251147d47dfbdf3a23acd3b5a5b 2023-04-22 19:55:32,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/1b07f80edceb4555a6a24a34ebbd4270 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1b07f80edceb4555a6a24a34ebbd4270 2023-04-22 19:55:32,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1b07f80edceb4555a6a24a34ebbd4270, entries=200, sequenceid=312, filesize=32.3 K 2023-04-22 19:55:32,513 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/56adcae4fba0413d9bd334d865f00407 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/56adcae4fba0413d9bd334d865f00407 2023-04-22 19:55:32,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:32,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193392506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/56adcae4fba0413d9bd334d865f00407, entries=150, sequenceid=312, filesize=11.9 K 2023-04-22 19:55:32,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/b9f1b251147d47dfbdf3a23acd3b5a5b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b9f1b251147d47dfbdf3a23acd3b5a5b 2023-04-22 19:55:32,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b9f1b251147d47dfbdf3a23acd3b5a5b, entries=150, sequenceid=312, filesize=11.9 K 2023-04-22 19:55:32,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~224.72 KB/230112, currentSize=100.63 KB/103050 for 75f361e1b049d8d7dacb8de0d0756852 in 523ms, sequenceid=312, compaction requested=true 2023-04-22 19:55:32,534 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:32,534 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:32,535 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-22 19:55:32,535 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-22 19:55:32,535 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:32,535 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:32,536 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:32,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:32,537 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 207866 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-22 19:55:32,537 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/A is initiating minor compaction (all files) 2023-04-22 19:55:32,537 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/A in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:32,537 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/267db742051340969cb9e3eaaf8f59db, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/237a506d606c472284e69e455daaf96f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/ddab1a4f68fe45f0b2ffba577c589614, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05e88652f92f4b6fb1cb5087390d6510, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/20b425791bcd47949501e1a1ad1b32fa, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/a19e6bc5e79b4fab9ca0727633222317, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/f993c41490df43cbab703638586b72c9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1b07f80edceb4555a6a24a34ebbd4270] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=203.0 K 2023-04-22 19:55:32,537 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 92160 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-22 19:55:32,537 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/B is initiating minor compaction (all files) 2023-04-22 19:55:32,537 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/B in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:32,538 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/9d85a8315d53481cb117a6f9408c29e8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/debf547051e9449a9a5fd9803eda09a9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2443fed9caf642a8879d0911d84636c7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/3cbdaadf1a674972961f2dc58787e63a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1bb1fb10e05743d79dd0e71fb1d2b611, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ac4b7d8414324bcca5bffbc323f1beb8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ee6a610c28444e82b4cc2c8f90841600, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/56adcae4fba0413d9bd334d865f00407] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=90 K 2023-04-22 19:55:32,538 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 267db742051340969cb9e3eaaf8f59db, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682193326865 2023-04-22 19:55:32,538 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 9d85a8315d53481cb117a6f9408c29e8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682193326865 2023-04-22 19:55:32,538 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 237a506d606c472284e69e455daaf96f, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193326903 2023-04-22 19:55:32,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:32,539 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting debf547051e9449a9a5fd9803eda09a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193326903 2023-04-22 19:55:32,539 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting ddab1a4f68fe45f0b2ffba577c589614, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682193327564 2023-04-22 19:55:32,539 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 2443fed9caf642a8879d0911d84636c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682193327564 2023-04-22 19:55:32,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:32,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:32,540 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:32,540 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:32,540 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:32,540 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:32,540 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,540 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,541 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 05e88652f92f4b6fb1cb5087390d6510, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682193328201 2023-04-22 19:55:32,541 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 3cbdaadf1a674972961f2dc58787e63a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682193328201 2023-04-22 19:55:32,541 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 20b425791bcd47949501e1a1ad1b32fa, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682193329812 2023-04-22 19:55:32,541 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1bb1fb10e05743d79dd0e71fb1d2b611, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682193329812 2023-04-22 19:55:32,541 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:32,541 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:32,542 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting a19e6bc5e79b4fab9ca0727633222317, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682193330200 2023-04-22 19:55:32,542 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:32,542 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:32,542 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ac4b7d8414324bcca5bffbc323f1beb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682193330200 2023-04-22 19:55:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,542 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting f993c41490df43cbab703638586b72c9, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682193331405 2023-04-22 19:55:32,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:32,542 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 1b07f80edceb4555a6a24a34ebbd4270, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682193331953 2023-04-22 19:55:32,542 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ee6a610c28444e82b4cc2c8f90841600, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682193331405 2023-04-22 19:55:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:32,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:32,543 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 56adcae4fba0413d9bd334d865f00407, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682193331963 2023-04-22 19:55:32,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:32,544 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:32,544 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:32,544 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:32,544 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 19:55:32,544 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:32,544 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:32,544 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:32,544 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:32,544 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:32,544 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:32,575 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#B#compaction#348 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:32,578 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422ce23013ce4534b8685949503c2a5b455 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ce23013ce4534b8685949503c2a5b455 2023-04-22 19:55:32,598 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#A#compaction#349 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:32,612 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=330, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/1bcbd36478a946b1a7389bf05bc258d0 2023-04-22 19:55:32,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:32,823 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:32,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:32,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193392913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,009 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/62ebcdc29c3048c1a96e76e4fd23b13f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/62ebcdc29c3048c1a96e76e4fd23b13f 2023-04-22 19:55:33,013 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/B of 75f361e1b049d8d7dacb8de0d0756852 into 62ebcdc29c3048c1a96e76e4fd23b13f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:33,013 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:33,013 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/B, priority=8, startTime=1682193332535; duration=0sec 2023-04-22 19:55:33,013 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:33,013 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-22 19:55:33,015 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 92160 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-22 19:55:33,015 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/C is initiating minor compaction (all files) 2023-04-22 19:55:33,015 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/C in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:33,015 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/4484a08068e944caab5569bfb3f973dd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/ec1a945b38ea4184bd299672b919875a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c17f7a9a6666468b843b36c39efbccfe, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/2ce019ff9ac14a0686665973ae844a4d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/77cc462ce6c6472f9249120a154246d2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/cd69669d5e184596882d968f456d46c0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9e93f186a42c4ba9853db8db4367afd4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b9f1b251147d47dfbdf3a23acd3b5a5b] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=90 K 2023-04-22 19:55:33,016 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 4484a08068e944caab5569bfb3f973dd, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682193326865 2023-04-22 19:55:33,016 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ec1a945b38ea4184bd299672b919875a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193326903 2023-04-22 19:55:33,016 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting c17f7a9a6666468b843b36c39efbccfe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682193327564 2023-04-22 19:55:33,017 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 2ce019ff9ac14a0686665973ae844a4d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682193328201 2023-04-22 19:55:33,017 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 77cc462ce6c6472f9249120a154246d2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682193329812 2023-04-22 19:55:33,017 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting cd69669d5e184596882d968f456d46c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682193330200 2023-04-22 19:55:33,018 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 9e93f186a42c4ba9853db8db4367afd4, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682193331405 2023-04-22 19:55:33,018 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b9f1b251147d47dfbdf3a23acd3b5a5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682193331963 2023-04-22 19:55:33,029 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#C#compaction#351 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:33,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:33,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193393024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,039 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/7cd36e6ef8f2408d9b601aacf517ffce 2023-04-22 19:55:33,053 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/f4d30f1a7f6a49e487460ae56eaa334a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/f4d30f1a7f6a49e487460ae56eaa334a 2023-04-22 19:55:33,056 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/A of 75f361e1b049d8d7dacb8de0d0756852 into f4d30f1a7f6a49e487460ae56eaa334a(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:33,057 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:33,057 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/A, priority=8, startTime=1682193332534; duration=0sec 2023-04-22 19:55:33,057 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:33,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:33,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193393231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,437 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/f98d86dfc7b042a686ad072afa190d6b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/f98d86dfc7b042a686ad072afa190d6b 2023-04-22 19:55:33,441 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/C of 75f361e1b049d8d7dacb8de0d0756852 into f98d86dfc7b042a686ad072afa190d6b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:33,441 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:33,441 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/C, priority=8, startTime=1682193332535; duration=0sec 2023-04-22 19:55:33,441 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:33,449 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/b204a77487fa402b8611412828bc42ed 2023-04-22 19:55:33,453 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/1bcbd36478a946b1a7389bf05bc258d0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1bcbd36478a946b1a7389bf05bc258d0 2023-04-22 19:55:33,456 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1bcbd36478a946b1a7389bf05bc258d0, entries=150, sequenceid=330, filesize=25.4 K 2023-04-22 19:55:33,457 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/7cd36e6ef8f2408d9b601aacf517ffce as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7cd36e6ef8f2408d9b601aacf517ffce 2023-04-22 19:55:33,462 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7cd36e6ef8f2408d9b601aacf517ffce, entries=150, sequenceid=330, filesize=11.9 K 2023-04-22 19:55:33,462 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/b204a77487fa402b8611412828bc42ed as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b204a77487fa402b8611412828bc42ed 2023-04-22 19:55:33,465 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b204a77487fa402b8611412828bc42ed, entries=150, sequenceid=330, filesize=11.9 K 2023-04-22 19:55:33,466 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 75f361e1b049d8d7dacb8de0d0756852 in 922ms, sequenceid=330, compaction requested=false 2023-04-22 19:55:33,466 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:33,466 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:33,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:33,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:33,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:33,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:33,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d7a21b8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:33,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:33,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,470 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:33,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:33,471 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:33,471 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,471 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:33,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:33,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:33,475 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:33,475 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:33,475 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:33,475 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:33,475 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:33,476 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,476 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,476 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,478 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:33,480 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:33,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:33,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:33,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:33,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:33,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,481 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,481 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,481 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,481 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:33,481 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:33,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:33,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:33,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:33,539 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 19:55:33,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:33,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:33,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:33,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:33,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:33,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:33,581 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:33,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:33,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:33,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:33,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:33,584 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:33,584 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:33,584 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:33,584 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:33,584 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,584 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:33,585 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:33,586 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,586 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:33,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:33,587 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:33,587 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:33,587 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:33,587 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:33,589 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,589 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,589 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,589 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:33,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a40e39[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:33,591 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:33,591 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,592 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:33,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:33,594 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:33,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:33,597 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:33,597 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:33,597 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:33,597 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:33,597 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:33,598 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,598 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,598 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:33,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:33,599 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,600 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,601 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,602 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,602 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,602 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:33,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:33,602 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:33,602 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:33,602 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:33,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:33,602 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:33,602 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,602 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:33,602 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:33,602 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:33,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:33,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193393621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,702 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:33,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:33,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:33,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:33,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:33,705 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:33,705 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:33,705 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:33,705 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:33,706 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,706 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,707 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:33,707 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:33,707 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,707 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:33,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:33,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:33,708 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:33,709 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:33,709 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:33,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:33,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:33,714 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:33,714 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:33,714 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:33,714 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:33,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:33,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:33,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:33,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:33,717 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,717 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,717 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,717 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:33,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e2d66[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:33,719 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:33,719 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,720 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:33,721 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,721 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:33,722 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:33,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:33,724 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:33,724 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:33,724 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:33,724 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:33,724 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:33,729 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,729 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,729 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:33,730 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,731 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,737 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:33,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:33,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:33,737 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:33,737 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:33,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:33,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193393731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,738 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:33,738 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:33,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:33,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:33,837 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:33,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:33,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:33,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:33,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:33,841 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:33,841 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:33,841 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:33,841 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:33,841 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,841 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,842 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:33,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:33,843 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,843 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:33,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:33,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:33,844 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:33,844 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:33,844 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:33,844 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:33,845 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,845 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,846 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,846 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:33,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77358056[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:33,847 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:33,847 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,849 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,849 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:33,849 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:33,849 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:33,849 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,849 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:33,850 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:33,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:33,853 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:33,853 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:33,853 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:33,853 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:33,853 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:33,854 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,854 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,854 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,854 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,856 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:33,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:33,858 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:33,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:33,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:33,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:33,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:33,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:33,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:33,859 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:33,859 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:33,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:33,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193393940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,951 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422e9c6ab83abea41ef99c4cf57a2df8f23 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e9c6ab83abea41ef99c4cf57a2df8f23 2023-04-22 19:55:33,959 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:33,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:33,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:33,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:33,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:33,962 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:33,962 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:33,962 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:33,962 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:33,963 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,963 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:33,965 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:33,966 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,966 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:33,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:33,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:33,967 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:33,967 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:33,967 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:33,968 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:33,968 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:33,968 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:33,968 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:33,970 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:33,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bee08e9[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:33,973 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:33,973 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,975 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,975 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:33,975 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:33,975 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:33,975 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,975 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:33,978 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:33,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:33,980 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:33,980 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:33,980 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:33,980 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:33,980 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:33,982 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,982 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:33,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,982 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:33,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:33,982 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:33,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,984 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,986 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:33,986 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:33,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:33,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:33,987 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:33,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:33,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:33,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:33,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:33,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,988 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:33,988 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:33,988 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:33,988 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:34,088 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:34,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:34,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:34,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:34,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:34,091 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:34,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:34,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:34,092 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,092 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,093 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,093 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,093 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,093 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:34,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:34,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:34,094 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:34,094 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,094 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:34,094 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:34,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:34,095 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,095 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:34,095 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:34,095 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:34,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:34,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:34,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:34,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,096 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,097 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,097 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:34,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1272106b[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:34,098 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:34,098 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,099 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:34,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:34,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:34,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,101 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:34,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:34,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:34,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:34,106 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:34,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:34,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:34,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:34,106 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:34,107 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,107 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,107 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,108 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,110 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,112 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,112 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,112 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:34,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:34,112 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,113 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,113 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,113 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,113 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,113 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:34,113 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:34,113 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,113 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,213 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:34,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:34,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:34,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:34,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:34,217 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:34,217 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,217 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:34,217 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:34,217 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,217 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,218 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,218 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,219 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,219 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:34,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:34,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:34,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:34,220 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,220 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:34,220 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:34,220 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:34,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:34,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63464a5e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:34,225 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:34,225 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,226 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:34,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:34,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:34,226 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,226 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:34,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:34,228 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:34,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:34,231 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:34,231 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:34,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:34,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:34,232 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:34,233 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,233 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,233 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,233 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,238 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,238 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,238 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,238 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:34,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:34,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,238 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:34,238 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,238 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:34,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:34,238 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:34,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193394245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,339 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:34,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:34,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:34,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:34,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:34,342 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:34,342 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,342 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:34,342 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:34,342 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,342 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,343 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,343 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:34,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:34,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:34,345 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,345 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:34,345 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:34,345 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:34,347 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:34,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382a4150[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:34,348 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:34,348 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,349 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:34,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:34,355 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:34,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:34,357 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:34,357 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:34,357 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:34,357 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:34,357 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:34,359 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,359 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,359 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,359 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,360 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/e35aca6c79be492594a9d77a9629153a 2023-04-22 19:55:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,362 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,364 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,364 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,364 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:34,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:34,364 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,364 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,365 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:34,365 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:34,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,465 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:34,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:34,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:34,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:34,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:34,468 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:34,468 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:34,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:34,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,470 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:34,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:34,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:34,472 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,472 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:34,472 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:34,472 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:34,474 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:34,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13a8affc[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:34,476 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:34,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,477 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,477 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:34,478 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:34,478 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:34,478 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,478 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:34,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:34,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:34,481 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:34,481 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:34,481 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:34,481 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:34,481 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:34,482 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,482 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,483 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,485 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:34,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:34,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,488 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,488 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:34,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,489 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,488 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:34,489 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,589 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:34,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:34,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:34,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:34,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:34,593 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:34,593 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,593 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:34,593 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:34,593 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,593 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,598 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:34,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:34,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:34,600 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:34,600 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,600 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:34,600 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:34,601 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,601 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:34,601 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:34,601 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:34,602 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,602 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:34,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11db14aa[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:34,604 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:34,604 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,606 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,606 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,606 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:34,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:34,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:34,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:34,611 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,611 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,611 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:34,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:34,614 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:34,614 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:34,614 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:34,614 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:34,614 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:34,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,616 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,617 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,618 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:34,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:34,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,619 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:34,619 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:34,619 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:34,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,619 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,719 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:34,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:34,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:34,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:34,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:34,723 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:34,723 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,723 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:34,723 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:34,724 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,724 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,725 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:34,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:34,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:34,727 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:34,727 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,727 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:34,727 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:34,727 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:34,728 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,728 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:34,728 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:34,728 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:34,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:34,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:34,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:34,732 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:34,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@409ea875[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:34,738 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:34,738 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,739 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,739 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:34,739 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:34,739 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:34,740 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,740 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:34,741 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:34,741 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,741 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,741 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:34,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:34,747 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:34,747 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:34,747 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:34,747 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:34,747 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:34,748 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,748 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,748 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,750 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,751 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:34,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:34,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,755 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,754 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,755 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,755 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:34,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,755 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,755 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:34,755 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,755 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:34,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193394749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/399e361250ff467e8caaa2bf1191c10f 2023-04-22 19:55:34,855 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:34,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:34,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:34,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:34,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:34,859 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:34,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:34,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:34,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,861 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:34,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:34,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:34,863 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,863 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:34,863 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:34,863 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:34,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ad451f7[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:34,866 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:34,867 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,868 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,868 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:34,868 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:34,868 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,868 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:34,868 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:34,869 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:34,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,869 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,869 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:34,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:34,872 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:34,872 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:34,872 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:34,872 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:34,872 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:34,874 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,874 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,874 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,874 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,876 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,878 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,878 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:34,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:34,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:34,878 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:34,878 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,878 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,878 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,878 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:34,878 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,878 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:34,878 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,978 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:34,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:34,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:34,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:34,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:34,983 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:34,983 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,983 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:34,983 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:34,984 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,984 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,985 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,985 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:34,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:34,985 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,985 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:34,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:34,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:34,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:34,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:34,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:34,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:34,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:34,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:34,987 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:34,990 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,990 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:34,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:34,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:34,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c55dc1e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:34,995 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:34,995 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:34,997 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:34,997 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,997 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:34,997 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:34,997 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:34,998 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,998 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:34,998 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:34,998 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:35,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:35,005 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:35,005 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:35,005 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:35,005 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:35,005 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:35,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,011 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,014 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,017 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,017 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,018 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,018 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,018 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,018 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:35,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:35,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:35,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:35,018 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,018 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,018 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,018 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,018 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:35,018 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:35,118 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:35,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:35,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:35,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:35,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:35,122 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:35,122 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,122 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:35,122 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:35,123 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,123 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,125 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:35,127 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,127 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:35,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:35,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:35,127 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,127 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:35,127 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:35,127 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,128 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:35,128 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:35,129 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:35,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3562f97a[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:35,131 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:35,131 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,132 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,133 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:35,133 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,133 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:35,133 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:35,133 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:35,137 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:35,139 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:35,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:35,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:35,139 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:35,139 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:35,141 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,141 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,141 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,152 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,155 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,155 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,155 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,155 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:35,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:35,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,155 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,156 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,155 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:35,156 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:35,156 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:35,156 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,156 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,156 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:35,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/a715f8ac1ed7482f96e7af4f170e75d6 2023-04-22 19:55:35,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/e35aca6c79be492594a9d77a9629153a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e35aca6c79be492594a9d77a9629153a 2023-04-22 19:55:35,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e35aca6c79be492594a9d77a9629153a, entries=200, sequenceid=348, filesize=32.3 K 2023-04-22 19:55:35,192 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/399e361250ff467e8caaa2bf1191c10f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/399e361250ff467e8caaa2bf1191c10f 2023-04-22 19:55:35,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/399e361250ff467e8caaa2bf1191c10f, entries=150, sequenceid=348, filesize=11.9 K 2023-04-22 19:55:35,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/a715f8ac1ed7482f96e7af4f170e75d6 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a715f8ac1ed7482f96e7af4f170e75d6 2023-04-22 19:55:35,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a715f8ac1ed7482f96e7af4f170e75d6, entries=150, sequenceid=348, filesize=11.9 K 2023-04-22 19:55:35,201 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 75f361e1b049d8d7dacb8de0d0756852 in 1663ms, sequenceid=348, compaction requested=true 2023-04-22 19:55:35,201 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:35,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:35,201 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:55:35,201 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:55:35,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:35,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:35,202 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:55:35,202 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85758 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:55:35,202 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/B is initiating minor compaction (all files) 2023-04-22 19:55:35,202 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/A is initiating minor compaction (all files) 2023-04-22 19:55:35,202 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/B in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,202 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/A in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,202 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/62ebcdc29c3048c1a96e76e4fd23b13f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7cd36e6ef8f2408d9b601aacf517ffce, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/399e361250ff467e8caaa2bf1191c10f] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=36.3 K 2023-04-22 19:55:35,202 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/f4d30f1a7f6a49e487460ae56eaa334a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1bcbd36478a946b1a7389bf05bc258d0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e35aca6c79be492594a9d77a9629153a] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=83.7 K 2023-04-22 19:55:35,203 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 62ebcdc29c3048c1a96e76e4fd23b13f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682193331963 2023-04-22 19:55:35,203 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting f4d30f1a7f6a49e487460ae56eaa334a, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682193331963 2023-04-22 19:55:35,203 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 7cd36e6ef8f2408d9b601aacf517ffce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682193332143 2023-04-22 19:55:35,203 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 1bcbd36478a946b1a7389bf05bc258d0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682193332143 2023-04-22 19:55:35,203 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 399e361250ff467e8caaa2bf1191c10f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682193332884 2023-04-22 19:55:35,204 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting e35aca6c79be492594a9d77a9629153a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682193332884 2023-04-22 19:55:35,217 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#B#compaction#357 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:35,226 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#A#compaction#356 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:35,256 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:35,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:35,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:35,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:35,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:35,259 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:35,259 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,259 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:35,259 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:35,259 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,260 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,261 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,261 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,261 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,261 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:35,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:35,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:35,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:35,264 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,264 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:35,269 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:35,269 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,269 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:35,269 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:35,269 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-22 19:55:35,269 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:35,269 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:35,269 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:35,269 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:35,269 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:35,269 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:35,294 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422d4df8b92b57c4f83aa4bd467826a6811 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d4df8b92b57c4f83aa4bd467826a6811 2023-04-22 19:55:35,301 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=365, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/aecce56f222746a49a614bfd1a5a30a2 2023-04-22 19:55:35,315 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/1b7fb611597b48ef89279a38ffe56110 2023-04-22 19:55:35,330 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/fa48724226e34d1482432b27e0e91d3b 2023-04-22 19:55:35,337 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/aecce56f222746a49a614bfd1a5a30a2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/aecce56f222746a49a614bfd1a5a30a2 2023-04-22 19:55:35,343 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/aecce56f222746a49a614bfd1a5a30a2, entries=150, sequenceid=365, filesize=25.4 K 2023-04-22 19:55:35,344 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/1b7fb611597b48ef89279a38ffe56110 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1b7fb611597b48ef89279a38ffe56110 2023-04-22 19:55:35,350 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1b7fb611597b48ef89279a38ffe56110, entries=150, sequenceid=365, filesize=11.9 K 2023-04-22 19:55:35,351 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/fa48724226e34d1482432b27e0e91d3b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/fa48724226e34d1482432b27e0e91d3b 2023-04-22 19:55:35,355 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/fa48724226e34d1482432b27e0e91d3b, entries=150, sequenceid=365, filesize=11.9 K 2023-04-22 19:55:35,356 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 75f361e1b049d8d7dacb8de0d0756852 in 87ms, sequenceid=365, compaction requested=true 2023-04-22 19:55:35,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:35,356 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:35,358 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,358 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,359 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,359 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:35,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8a32a3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:35,363 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:35,363 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,364 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,364 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:35,364 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:35,364 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:35,364 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,364 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:35,366 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:35,366 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,366 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:35,366 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:35,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:35,374 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:35,375 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:35,375 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:35,375 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:35,375 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:35,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,376 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,376 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,378 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,380 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,380 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:35,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,380 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:35,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:35,380 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,380 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,380 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,380 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,380 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,381 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,381 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,380 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:35,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,381 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:35,486 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:35,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:35,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:35,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:35,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:35,490 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:35,490 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,490 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:35,490 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:35,491 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,491 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,493 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,493 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,493 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:35,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:35,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:35,495 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:35,495 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,495 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:35,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:35,508 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:35,508 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:35,508 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:35,508 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:35,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:35,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,508 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:35,509 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:35,510 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,510 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,510 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,510 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:35,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d990ac[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:35,512 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:35,512 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,513 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:35,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:35,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:35,514 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,514 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:35,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:35,524 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:35,524 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:35,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:35,527 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:35,527 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:35,527 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:35,527 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:35,527 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:35,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,530 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,531 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,534 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:35,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:35,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:35,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,535 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:35,535 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:35,630 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/0c45922c7bf149668a371ff167d988cc as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/0c45922c7bf149668a371ff167d988cc 2023-04-22 19:55:35,635 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/B of 75f361e1b049d8d7dacb8de0d0756852 into 0c45922c7bf149668a371ff167d988cc(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:35,635 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:35,635 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/B, priority=13, startTime=1682193335201; duration=0sec 2023-04-22 19:55:35,635 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:35,635 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:35,635 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:55:35,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:35,637 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:55:35,637 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/C is initiating minor compaction (all files) 2023-04-22 19:55:35,637 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/C in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,637 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/f98d86dfc7b042a686ad072afa190d6b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b204a77487fa402b8611412828bc42ed, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a715f8ac1ed7482f96e7af4f170e75d6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/fa48724226e34d1482432b27e0e91d3b] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=48.2 K 2023-04-22 19:55:35,637 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f98d86dfc7b042a686ad072afa190d6b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682193331963 2023-04-22 19:55:35,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:35,638 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b204a77487fa402b8611412828bc42ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682193332143 2023-04-22 19:55:35,638 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a715f8ac1ed7482f96e7af4f170e75d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682193332884 2023-04-22 19:55:35,638 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting fa48724226e34d1482432b27e0e91d3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1682193333607 2023-04-22 19:55:35,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:35,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:35,644 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:35,644 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,644 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:35,644 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:35,645 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,645 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:35,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:35,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:35,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:35,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:35,649 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#C#compaction#361 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:35,664 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:35,664 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:35,674 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/88ec84e2b4c14b628f18dce9633687a5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/88ec84e2b4c14b628f18dce9633687a5 2023-04-22 19:55:35,676 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,676 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:35,676 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:35,676 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,676 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:35,676 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:35,676 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,679 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/A of 75f361e1b049d8d7dacb8de0d0756852 into 88ec84e2b4c14b628f18dce9633687a5(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:35,679 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:35,679 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/A, priority=13, startTime=1682193335201; duration=0sec 2023-04-22 19:55:35,679 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:35,680 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:35,680 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:35,683 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,683 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a3e1885[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:35,686 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:35,686 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,687 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:35,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:35,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:35,688 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/bef44c9b53b14293ad17b1b6555a8802 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/bef44c9b53b14293ad17b1b6555a8802 2023-04-22 19:55:35,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:35,689 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:35,689 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:35,689 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:35,689 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,692 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/C of 75f361e1b049d8d7dacb8de0d0756852 into bef44c9b53b14293ad17b1b6555a8802(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:35,692 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:35,692 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/C, priority=12, startTime=1682193335201; duration=0sec 2023-04-22 19:55:35,692 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:35,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:35,693 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:35,693 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:35,693 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:35,693 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:35,693 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:35,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,695 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,697 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,700 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:35,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:35,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:35,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,701 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:35,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,701 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:35,802 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:35,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:35,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:35,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:35,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:35,809 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:35,809 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,809 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:35,809 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:35,810 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,810 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,811 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,811 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,812 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,812 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:35,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:35,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:35,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:35,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:35,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:35,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:35,816 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,816 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:35,817 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:35,817 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:35,818 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,818 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:35,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78358391[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:35,821 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:35,821 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,822 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,823 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:35,823 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,823 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:35,823 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:35,823 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:35,825 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:35,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:35,829 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:35,829 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:35,829 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:35,829 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:35,829 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:35,832 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:35,832 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,832 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:35,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,833 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:35,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,834 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,838 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,838 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,838 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:35,838 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:35,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:35,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,838 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,839 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,839 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,839 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,839 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:35,839 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,839 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:35,839 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:35,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:35,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:35,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:35,939 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:35,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:35,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:35,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:35,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:35,950 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:35,950 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,951 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:35,951 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:35,951 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,951 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,952 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,952 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:35,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:35,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:35,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:35,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:35,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:35,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:35,954 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:35,954 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:35,954 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:35,964 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:35,964 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:35,964 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:35,964 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:35,964 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-22 19:55:35,966 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:35,966 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:35,966 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:35,966 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:35,966 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:35,966 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:36,003 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304224ccd626995264ea7bc268420fedd54c4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224ccd626995264ea7bc268420fedd54c4 2023-04-22 19:55:36,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:36,010 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:36,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193336063, max timestamp=1682193336088Num uniques 859; 2023-04-22 19:55:36,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193336063, max timestamp=1682193336088Num uniques 859; 2023-04-22 19:55:36,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193336063, max timestamp=1682193336088Num uniques 859; 2023-04-22 19:55:36,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:36,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:36,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:55:36,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193336171, max timestamp=1682193336221Num uniques 1018; 2023-04-22 19:55:36,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 19:55:36,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193336171, max timestamp=1682193336221Num uniques 1018; 2023-04-22 19:55:36,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193336171, max timestamp=1682193336221Num uniques 1018; 2023-04-22 19:55:36,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 19:55:36,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 19:55:36,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682193336296, max timestamp=1682193336313Num uniques 1027; 2023-04-22 19:55:36,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 19:55:36,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682193336296, max timestamp=1682193336313Num uniques 1027; 2023-04-22 19:55:36,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 19:55:36,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682193336296, max timestamp=1682193336313Num uniques 1027; 2023-04-22 19:55:36,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 19:55:36,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682193336377, max timestamp=1682193336400Num uniques 1036; 2023-04-22 19:55:36,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-22 19:55:36,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682193336377, max timestamp=1682193336400Num uniques 1036; 2023-04-22 19:55:36,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682193336377, max timestamp=1682193336400Num uniques 1036; 2023-04-22 19:55:36,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-22 19:55:36,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-22 19:55:36,430 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=379, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/70867959e8d84f95a7641f9c067f5e88 2023-04-22 19:55:36,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193396429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193396430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193396431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193396432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193396435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193396537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193396540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193396541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193396541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193396543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193396745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193396745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193396745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193396746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:36,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193396751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:36,853 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/d8d2f63e5aef41d599cbb4684b8479cb 2023-04-22 19:55:37,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193397049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193397051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193397051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193397053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193397056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,263 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/9dc376fbce7847bebc0db0190e3cf2f4 2023-04-22 19:55:37,268 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/70867959e8d84f95a7641f9c067f5e88 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/70867959e8d84f95a7641f9c067f5e88 2023-04-22 19:55:37,273 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/70867959e8d84f95a7641f9c067f5e88, entries=150, sequenceid=379, filesize=25.4 K 2023-04-22 19:55:37,273 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/d8d2f63e5aef41d599cbb4684b8479cb as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/d8d2f63e5aef41d599cbb4684b8479cb 2023-04-22 19:55:37,281 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/d8d2f63e5aef41d599cbb4684b8479cb, entries=100, sequenceid=379, filesize=9.5 K 2023-04-22 19:55:37,281 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/9dc376fbce7847bebc0db0190e3cf2f4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9dc376fbce7847bebc0db0190e3cf2f4 2023-04-22 19:55:37,287 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9dc376fbce7847bebc0db0190e3cf2f4, entries=100, sequenceid=379, filesize=9.5 K 2023-04-22 19:55:37,288 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 75f361e1b049d8d7dacb8de0d0756852 in 1324ms, sequenceid=379, compaction requested=true 2023-04-22 19:55:37,288 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:37,288 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:37,288 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:37,288 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:37,288 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,288 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:37,288 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:37,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:37,291 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:37,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:37,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29fbb91e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:37,293 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:37,293 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:37,294 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:37,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:37,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:37,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:37,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:37,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:37,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:37,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:37,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:37,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:37,305 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:37,308 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:37,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:37,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:37,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:37,308 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:37,313 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:37,313 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:37,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:37,314 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:37,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:37,315 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:37,316 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,318 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:37,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:37,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:37,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:37,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:37,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:37,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:37,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:37,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:37,319 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:37,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:37,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:37,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:37,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:37,319 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:37,319 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:37,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:37,420 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:37,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:37,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:37,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:37,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:37,424 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:37,424 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:37,424 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:37,424 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:37,425 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:37,425 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:37,426 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,426 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:37,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:37,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:37,426 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,426 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:37,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:37,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:37,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:37,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:37,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:37,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:37,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:37,427 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:37,427 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:37,427 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:37,427 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:37,427 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:37,427 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:37,427 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:37,427 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=147.60 KB heapSize=412.92 KB 2023-04-22 19:55:37,428 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:37,428 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:37,428 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:37,428 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:37,428 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:37,428 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:37,511 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-22 19:55:37,571 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e6662d791b874985b818ef99d61f35dd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/bbfe820aa13a4cffbe1a2405b9b112eb, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/b69250230ba142328895f47031187ff4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/0dfc773a18df4d30a8fc4b45a42736bd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/8d48bcfb1e714f098b5ffd08dcfbac14, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/dd7161ce3114408990893c1b38e42951, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/52c4c901d49744f9ba4d088ccf2f9a30, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05c1d3ecde964872828d46b6f31c6d47, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/57360678ac094725bf6639cdd49bae34, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/267db742051340969cb9e3eaaf8f59db, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/237a506d606c472284e69e455daaf96f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/ddab1a4f68fe45f0b2ffba577c589614, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05e88652f92f4b6fb1cb5087390d6510, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/20b425791bcd47949501e1a1ad1b32fa, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/a19e6bc5e79b4fab9ca0727633222317, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/f993c41490df43cbab703638586b72c9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1b07f80edceb4555a6a24a34ebbd4270, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/f4d30f1a7f6a49e487460ae56eaa334a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1bcbd36478a946b1a7389bf05bc258d0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e35aca6c79be492594a9d77a9629153a] to archive 2023-04-22 19:55:37,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:37,573 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:37,573 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:55:37,582 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e6662d791b874985b818ef99d61f35dd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e6662d791b874985b818ef99d61f35dd 2023-04-22 19:55:37,583 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/bbfe820aa13a4cffbe1a2405b9b112eb to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/bbfe820aa13a4cffbe1a2405b9b112eb 2023-04-22 19:55:37,589 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/b69250230ba142328895f47031187ff4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/b69250230ba142328895f47031187ff4 2023-04-22 19:55:37,590 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/0dfc773a18df4d30a8fc4b45a42736bd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/0dfc773a18df4d30a8fc4b45a42736bd 2023-04-22 19:55:37,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193397573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,592 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/8d48bcfb1e714f098b5ffd08dcfbac14 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/8d48bcfb1e714f098b5ffd08dcfbac14 2023-04-22 19:55:37,593 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/dd7161ce3114408990893c1b38e42951 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/dd7161ce3114408990893c1b38e42951 2023-04-22 19:55:37,594 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/52c4c901d49744f9ba4d088ccf2f9a30 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/52c4c901d49744f9ba4d088ccf2f9a30 2023-04-22 19:55:37,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193397581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193397582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193397582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193397585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,596 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05c1d3ecde964872828d46b6f31c6d47 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05c1d3ecde964872828d46b6f31c6d47 2023-04-22 19:55:37,597 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/57360678ac094725bf6639cdd49bae34 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/57360678ac094725bf6639cdd49bae34 2023-04-22 19:55:37,598 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/267db742051340969cb9e3eaaf8f59db to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/267db742051340969cb9e3eaaf8f59db 2023-04-22 19:55:37,600 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/237a506d606c472284e69e455daaf96f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/237a506d606c472284e69e455daaf96f 2023-04-22 19:55:37,601 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/ddab1a4f68fe45f0b2ffba577c589614 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/ddab1a4f68fe45f0b2ffba577c589614 2023-04-22 19:55:37,602 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05e88652f92f4b6fb1cb5087390d6510 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/05e88652f92f4b6fb1cb5087390d6510 2023-04-22 19:55:37,603 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/20b425791bcd47949501e1a1ad1b32fa to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/20b425791bcd47949501e1a1ad1b32fa 2023-04-22 19:55:37,605 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/a19e6bc5e79b4fab9ca0727633222317 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/a19e6bc5e79b4fab9ca0727633222317 2023-04-22 19:55:37,606 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/f993c41490df43cbab703638586b72c9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/f993c41490df43cbab703638586b72c9 2023-04-22 19:55:37,607 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1b07f80edceb4555a6a24a34ebbd4270 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1b07f80edceb4555a6a24a34ebbd4270 2023-04-22 19:55:37,608 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/f4d30f1a7f6a49e487460ae56eaa334a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/f4d30f1a7f6a49e487460ae56eaa334a 2023-04-22 19:55:37,609 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1bcbd36478a946b1a7389bf05bc258d0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1bcbd36478a946b1a7389bf05bc258d0 2023-04-22 19:55:37,611 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e35aca6c79be492594a9d77a9629153a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/e35aca6c79be492594a9d77a9629153a 2023-04-22 19:55:37,618 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/20d3fc783d9e466fa6cabfe6ef3b51af, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2122305e7f6a4746a61ef497721500d7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/4d4a8e0265764a7b8b164e0ea834cf2b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/667d3f4b75724394909fb323978eef40, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/55546ed725c944bd9dd9e449198aa403, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/da840760ad024b5a8919672627408cdd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ebe0f36fdb44461591dda7bd3ac402ba, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/011a3638535c466fb9b99de651bc1abb, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/9d85a8315d53481cb117a6f9408c29e8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/6fb23b3fd06040cd8631671ed422f0c3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/debf547051e9449a9a5fd9803eda09a9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2443fed9caf642a8879d0911d84636c7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/3cbdaadf1a674972961f2dc58787e63a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1bb1fb10e05743d79dd0e71fb1d2b611, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ac4b7d8414324bcca5bffbc323f1beb8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ee6a610c28444e82b4cc2c8f90841600, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/62ebcdc29c3048c1a96e76e4fd23b13f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/56adcae4fba0413d9bd334d865f00407, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7cd36e6ef8f2408d9b601aacf517ffce, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/399e361250ff467e8caaa2bf1191c10f] to archive 2023-04-22 19:55:37,619 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:55:37,620 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/20d3fc783d9e466fa6cabfe6ef3b51af to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/20d3fc783d9e466fa6cabfe6ef3b51af 2023-04-22 19:55:37,622 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2122305e7f6a4746a61ef497721500d7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2122305e7f6a4746a61ef497721500d7 2023-04-22 19:55:37,623 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/4d4a8e0265764a7b8b164e0ea834cf2b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/4d4a8e0265764a7b8b164e0ea834cf2b 2023-04-22 19:55:37,625 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/667d3f4b75724394909fb323978eef40 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/667d3f4b75724394909fb323978eef40 2023-04-22 19:55:37,626 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/55546ed725c944bd9dd9e449198aa403 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/55546ed725c944bd9dd9e449198aa403 2023-04-22 19:55:37,636 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/da840760ad024b5a8919672627408cdd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/da840760ad024b5a8919672627408cdd 2023-04-22 19:55:37,638 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ebe0f36fdb44461591dda7bd3ac402ba to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ebe0f36fdb44461591dda7bd3ac402ba 2023-04-22 19:55:37,639 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/011a3638535c466fb9b99de651bc1abb to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/011a3638535c466fb9b99de651bc1abb 2023-04-22 19:55:37,640 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/9d85a8315d53481cb117a6f9408c29e8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/9d85a8315d53481cb117a6f9408c29e8 2023-04-22 19:55:37,641 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/6fb23b3fd06040cd8631671ed422f0c3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/6fb23b3fd06040cd8631671ed422f0c3 2023-04-22 19:55:37,642 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/debf547051e9449a9a5fd9803eda09a9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/debf547051e9449a9a5fd9803eda09a9 2023-04-22 19:55:37,643 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2443fed9caf642a8879d0911d84636c7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2443fed9caf642a8879d0911d84636c7 2023-04-22 19:55:37,645 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/3cbdaadf1a674972961f2dc58787e63a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/3cbdaadf1a674972961f2dc58787e63a 2023-04-22 19:55:37,648 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1bb1fb10e05743d79dd0e71fb1d2b611 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1bb1fb10e05743d79dd0e71fb1d2b611 2023-04-22 19:55:37,649 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ac4b7d8414324bcca5bffbc323f1beb8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ac4b7d8414324bcca5bffbc323f1beb8 2023-04-22 19:55:37,650 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ee6a610c28444e82b4cc2c8f90841600 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/ee6a610c28444e82b4cc2c8f90841600 2023-04-22 19:55:37,652 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/62ebcdc29c3048c1a96e76e4fd23b13f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/62ebcdc29c3048c1a96e76e4fd23b13f 2023-04-22 19:55:37,653 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/56adcae4fba0413d9bd334d865f00407 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/56adcae4fba0413d9bd334d865f00407 2023-04-22 19:55:37,654 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7cd36e6ef8f2408d9b601aacf517ffce to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7cd36e6ef8f2408d9b601aacf517ffce 2023-04-22 19:55:37,656 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/399e361250ff467e8caaa2bf1191c10f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/399e361250ff467e8caaa2bf1191c10f 2023-04-22 19:55:37,661 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/bde4c0f55b5c4cb09d545b787ac167c3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/4aa6d108e6654049ac35e700490da289, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/1a251e930369436b816ca050a2c2a8bd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/62cb16ea76db448cae1e5cda6a9db99c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/6bee8f55a2934851abeaa012cfe13c6b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/dc7b86a2cdb7437b9638e467a5fa2b21, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b049af2203144df39ea931e236b9391f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/f62a306d0bde4c81b0e9a235e20109ba, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/4484a08068e944caab5569bfb3f973dd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/7b128e16d326445dadccefc7fe142c61, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/ec1a945b38ea4184bd299672b919875a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c17f7a9a6666468b843b36c39efbccfe, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/2ce019ff9ac14a0686665973ae844a4d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/77cc462ce6c6472f9249120a154246d2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/cd69669d5e184596882d968f456d46c0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9e93f186a42c4ba9853db8db4367afd4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/f98d86dfc7b042a686ad072afa190d6b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b9f1b251147d47dfbdf3a23acd3b5a5b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b204a77487fa402b8611412828bc42ed, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a715f8ac1ed7482f96e7af4f170e75d6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/fa48724226e34d1482432b27e0e91d3b] to archive 2023-04-22 19:55:37,661 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:55:37,663 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/bde4c0f55b5c4cb09d545b787ac167c3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/bde4c0f55b5c4cb09d545b787ac167c3 2023-04-22 19:55:37,665 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/4aa6d108e6654049ac35e700490da289 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/4aa6d108e6654049ac35e700490da289 2023-04-22 19:55:37,666 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/1a251e930369436b816ca050a2c2a8bd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/1a251e930369436b816ca050a2c2a8bd 2023-04-22 19:55:37,669 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/62cb16ea76db448cae1e5cda6a9db99c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/62cb16ea76db448cae1e5cda6a9db99c 2023-04-22 19:55:37,671 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/6bee8f55a2934851abeaa012cfe13c6b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/6bee8f55a2934851abeaa012cfe13c6b 2023-04-22 19:55:37,674 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/dc7b86a2cdb7437b9638e467a5fa2b21 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/dc7b86a2cdb7437b9638e467a5fa2b21 2023-04-22 19:55:37,676 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b049af2203144df39ea931e236b9391f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b049af2203144df39ea931e236b9391f 2023-04-22 19:55:37,677 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/f62a306d0bde4c81b0e9a235e20109ba to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/f62a306d0bde4c81b0e9a235e20109ba 2023-04-22 19:55:37,681 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/4484a08068e944caab5569bfb3f973dd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/4484a08068e944caab5569bfb3f973dd 2023-04-22 19:55:37,690 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/7b128e16d326445dadccefc7fe142c61 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/7b128e16d326445dadccefc7fe142c61 2023-04-22 19:55:37,691 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/ec1a945b38ea4184bd299672b919875a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/ec1a945b38ea4184bd299672b919875a 2023-04-22 19:55:37,693 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c17f7a9a6666468b843b36c39efbccfe to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c17f7a9a6666468b843b36c39efbccfe 2023-04-22 19:55:37,694 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/2ce019ff9ac14a0686665973ae844a4d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/2ce019ff9ac14a0686665973ae844a4d 2023-04-22 19:55:37,696 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/77cc462ce6c6472f9249120a154246d2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/77cc462ce6c6472f9249120a154246d2 2023-04-22 19:55:37,698 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/cd69669d5e184596882d968f456d46c0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/cd69669d5e184596882d968f456d46c0 2023-04-22 19:55:37,700 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9e93f186a42c4ba9853db8db4367afd4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9e93f186a42c4ba9853db8db4367afd4 2023-04-22 19:55:37,701 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/f98d86dfc7b042a686ad072afa190d6b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/f98d86dfc7b042a686ad072afa190d6b 2023-04-22 19:55:37,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40330 deadline: 1682193397693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40320 deadline: 1682193397696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,702 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b9f1b251147d47dfbdf3a23acd3b5a5b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b9f1b251147d47dfbdf3a23acd3b5a5b 2023-04-22 19:55:37,703 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b204a77487fa402b8611412828bc42ed to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/b204a77487fa402b8611412828bc42ed 2023-04-22 19:55:37,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,705 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a715f8ac1ed7482f96e7af4f170e75d6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a715f8ac1ed7482f96e7af4f170e75d6 2023-04-22 19:55:37,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193397697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193397697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193397697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,707 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/fa48724226e34d1482432b27e0e91d3b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/fa48724226e34d1482432b27e0e91d3b 2023-04-22 19:55:37,862 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422b3650c7beef749bab68408bc6e0492b4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b3650c7beef749bab68408bc6e0492b4 2023-04-22 19:55:37,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40330 deadline: 1682193397903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40320 deadline: 1682193397904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193397906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40278 deadline: 1682193397906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:37,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:37,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193397907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40278 deadline: 1682193398211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40320 deadline: 1682193398211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40330 deadline: 1682193398211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40342 deadline: 1682193398212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:38,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193398212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,270 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=467, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/d4b120326aae42afa663ee40c6161fa1 2023-04-22 19:55:38,686 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/7c2d2c68a2b8472bb095fb65107ebc90 2023-04-22 19:55:38,710 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/a9f2988d9c9741b78ba92f401e7ca8e7 2023-04-22 19:55:38,717 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/d4b120326aae42afa663ee40c6161fa1 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/d4b120326aae42afa663ee40c6161fa1 2023-04-22 19:55:38,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:38,725 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/d4b120326aae42afa663ee40c6161fa1, entries=150, sequenceid=467, filesize=25.4 K 2023-04-22 19:55:38,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40320 deadline: 1682193398716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:38,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40342 deadline: 1682193398716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:38,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40330 deadline: 1682193398716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,726 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/7c2d2c68a2b8472bb095fb65107ebc90 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7c2d2c68a2b8472bb095fb65107ebc90 2023-04-22 19:55:38,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:38,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40278 deadline: 1682193398717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:38,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193398722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,732 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7c2d2c68a2b8472bb095fb65107ebc90, entries=150, sequenceid=467, filesize=11.9 K 2023-04-22 19:55:38,733 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/a9f2988d9c9741b78ba92f401e7ca8e7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a9f2988d9c9741b78ba92f401e7ca8e7 2023-04-22 19:55:38,739 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a9f2988d9c9741b78ba92f401e7ca8e7, entries=150, sequenceid=467, filesize=11.9 K 2023-04-22 19:55:38,740 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~412.88 KB/422784, currentSize=33.54 KB/34350 for 75f361e1b049d8d7dacb8de0d0756852 in 1313ms, sequenceid=467, compaction requested=true 2023-04-22 19:55:38,741 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:38,741 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:38,741 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:38,741 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:38,741 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,741 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:38,741 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:38,742 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:38,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:38,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:38,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:38,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:38,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6de3007d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:38,747 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:38,747 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:38,750 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:38,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:38,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:38,750 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:38,750 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:38,750 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:38,750 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,750 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:38,752 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,752 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:38,752 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:38,752 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:38,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:38,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:38,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:38,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:38,755 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:38,755 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:38,755 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:38,755 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:38,755 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:38,757 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:38,757 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:38,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:38,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:38,757 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:38,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:38,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:38,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:38,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,759 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:38,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:38,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:38,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:38,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:38,761 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:38,761 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:38,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:38,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:38,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:38,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:38,762 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,762 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:38,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:38,762 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:38,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:38,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:38,762 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:38,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:38,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:38,763 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:38,763 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:38,763 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:38,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:38,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:38,863 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:38,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:38,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:38,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:38,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:38,876 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:38,876 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:38,876 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:38,876 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:38,877 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:38,877 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:38,878 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:38,878 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:38,878 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:38,878 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:38,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:38,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:38,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:38,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:38,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:38,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:38,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:38,879 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:38,880 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:38,880 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-22 19:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:38,911 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304227f3769bb13ed4e698a00762fb18f9074 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304227f3769bb13ed4e698a00762fb18f9074 2023-04-22 19:55:38,926 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=475, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/1444aadcb0664686b1e79c0463f72391 2023-04-22 19:55:39,350 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/9a88bc6c636242b082cdf8e4d6131e74 2023-04-22 19:55:39,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:39,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:39,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193339749, max timestamp=1682193339764Num uniques 859; 2023-04-22 19:55:39,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:55:39,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193339749, max timestamp=1682193339764Num uniques 859; 2023-04-22 19:55:39,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193339749, max timestamp=1682193339764Num uniques 859; 2023-04-22 19:55:39,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:55:39,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:55:39,778 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/af1b1dd4193847849ad6e063ae272b43 2023-04-22 19:55:39,782 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/1444aadcb0664686b1e79c0463f72391 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1444aadcb0664686b1e79c0463f72391 2023-04-22 19:55:39,785 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1444aadcb0664686b1e79c0463f72391, entries=150, sequenceid=475, filesize=25.4 K 2023-04-22 19:55:39,786 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/9a88bc6c636242b082cdf8e4d6131e74 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/9a88bc6c636242b082cdf8e4d6131e74 2023-04-22 19:55:39,790 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/9a88bc6c636242b082cdf8e4d6131e74, entries=150, sequenceid=475, filesize=11.9 K 2023-04-22 19:55:39,790 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/af1b1dd4193847849ad6e063ae272b43 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/af1b1dd4193847849ad6e063ae272b43 2023-04-22 19:55:39,793 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/af1b1dd4193847849ad6e063ae272b43, entries=150, sequenceid=475, filesize=11.9 K 2023-04-22 19:55:39,795 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=80.51 KB/82440 for 75f361e1b049d8d7dacb8de0d0756852 in 915ms, sequenceid=475, compaction requested=true 2023-04-22 19:55:39,795 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:39,795 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:39,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:39,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:39,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:39,795 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:39,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:39,796 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=93.93 KB heapSize=256.08 KB 2023-04-22 19:55:39,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:39,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:39,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:39,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:39,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:39,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:39,801 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,801 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:39,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,802 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:39,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:39,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4733d774[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:39,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:39,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,805 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,805 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:39,805 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:39,805 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:39,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:39,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:39,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:39,808 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,808 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:39,809 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:39,809 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:39,809 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:39,809 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:39,809 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:39,809 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:39,812 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,812 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:39,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,813 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:39,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:39,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:39,819 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:39,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,829 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:39,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:39,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,831 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,831 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:39,831 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:39,831 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:39,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:39,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:39,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:39,831 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:39,831 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:39,832 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:39,831 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,832 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:39,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:39,832 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,832 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,832 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:39,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193399866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:39,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40330 deadline: 1682193399867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:39,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40278 deadline: 1682193399868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:39,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40342 deadline: 1682193399869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:39,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40320 deadline: 1682193399876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,932 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:39,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:39,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:39,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:39,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:39,936 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:39,936 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:39,936 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:39,936 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:39,936 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,936 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:39,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,937 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:39,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:39,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:39,938 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,938 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:39,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:39,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:39,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:39,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:39,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:39,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:39,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:39,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:39,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:39,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:39,939 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:39,939 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:39,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:39,939 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:39,939 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:39,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:39,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:39,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:39,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:39,942 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:39,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:39,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:39,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13ceed08[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:39,944 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:39,944 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,945 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:39,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:39,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:39,945 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,945 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:39,946 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:39,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:39,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:39,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:39,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:39,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:39,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:39,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:39,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:39,950 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:39,950 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:39,950 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:39,950 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:39,950 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:39,952 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:39,952 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,952 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:39,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:39,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,954 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:39,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:39,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:39,965 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:39,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:39,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:39,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:39,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:39,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:39,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:39,966 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:39,966 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:39,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:39,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:39,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:39,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193399979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:39,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40278 deadline: 1682193399979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:39,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40330 deadline: 1682193399980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:39,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40342 deadline: 1682193399981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:39,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:39,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40320 deadline: 1682193399987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,066 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:40,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:40,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:40,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:40,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:40,069 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:40,069 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:40,069 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:40,069 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:40,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:40,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:40,071 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:40,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:40,071 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:40,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:40,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:40,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:40,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:40,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:40,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:40,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:40,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:40,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. started... 2023-04-22 19:55:40,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. as already flushing 2023-04-22 19:55:40,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:40,078 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:40,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:40,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:40,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67899526[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:40,084 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:40,084 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:40,094 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,094 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:40,095 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:40,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:40,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:40,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:40,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:40,096 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:40,096 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:40,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:40,096 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:40,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:40,099 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:40,099 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:40,099 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:40,099 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:40,099 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:40,100 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:40,100 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:40,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:40,100 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:40,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:40,102 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:40,103 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,105 DEBUG [Thread-2253] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06a0df34 to 127.0.0.1:53381 2023-04-22 19:55:40,105 DEBUG [Thread-2253] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:40,106 DEBUG [Thread-2259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e86d9 to 127.0.0.1:53381 2023-04-22 19:55:40,106 DEBUG [Thread-2259] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:40,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:40,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:40,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:40,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:40,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:40,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:40,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:40,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:40,107 DEBUG [Thread-2250] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:40,107 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:40,107 DEBUG [Thread-2250] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:40,107 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:40,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:40,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:40,108 DEBUG [Thread-2251] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:53381 2023-04-22 19:55:40,108 DEBUG [Thread-2251] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:40,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:40,121 DEBUG [Thread-2257] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21128196 to 127.0.0.1:53381 2023-04-22 19:55:40,121 DEBUG [Thread-2257] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:40,121 DEBUG [Thread-2255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:53381 2023-04-22 19:55:40,121 DEBUG [Thread-2255] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:40,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40278 deadline: 1682193400182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193400182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40330 deadline: 1682193400183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40342 deadline: 1682193400190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40320 deadline: 1682193400196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,207 DEBUG [Thread-2250] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:40,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:40,229 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422a632efa07ece4a2781ada8989e7f2b97 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a632efa07ece4a2781ada8989e7f2b97 2023-04-22 19:55:40,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193400484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40278 deadline: 1682193400485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40330 deadline: 1682193400486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40342 deadline: 1682193400493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40320 deadline: 1682193400497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,634 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=505, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/276fb6d14a4040c08f82706656eaa11e 2023-04-22 19:55:40,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40278 deadline: 1682193400986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40330 deadline: 1682193400988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40310 deadline: 1682193400990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:40,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:40,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40342 deadline: 1682193400996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:41,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:41,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40320 deadline: 1682193401001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:41,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/2c2f2dca306e49d696ebd08635334d80 2023-04-22 19:55:41,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/d4bda3df63224615b4bfb9ee5ff932ce 2023-04-22 19:55:41,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/276fb6d14a4040c08f82706656eaa11e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/276fb6d14a4040c08f82706656eaa11e 2023-04-22 19:55:41,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/276fb6d14a4040c08f82706656eaa11e, entries=200, sequenceid=505, filesize=32.3 K 2023-04-22 19:55:41,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/2c2f2dca306e49d696ebd08635334d80 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2c2f2dca306e49d696ebd08635334d80 2023-04-22 19:55:41,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2c2f2dca306e49d696ebd08635334d80, entries=150, sequenceid=505, filesize=11.9 K 2023-04-22 19:55:41,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/d4bda3df63224615b4bfb9ee5ff932ce as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/d4bda3df63224615b4bfb9ee5ff932ce 2023-04-22 19:55:41,465 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/d4bda3df63224615b4bfb9ee5ff932ce, entries=150, sequenceid=505, filesize=11.9 K 2023-04-22 19:55:41,466 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~275.95 KB/282576, currentSize=87.22 KB/89310 for 75f361e1b049d8d7dacb8de0d0756852 in 1670ms, sequenceid=505, compaction requested=true 2023-04-22 19:55:41,466 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:41,466 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:41,466 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:41,466 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 19:55:41,466 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 19:55:41,466 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:41,468 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163994 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 19:55:41,468 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71354 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 19:55:41,468 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/A is initiating minor compaction (all files) 2023-04-22 19:55:41,468 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/B is initiating minor compaction (all files) 2023-04-22 19:55:41,468 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/A in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:41,468 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/B in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:41,468 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/88ec84e2b4c14b628f18dce9633687a5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/aecce56f222746a49a614bfd1a5a30a2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/70867959e8d84f95a7641f9c067f5e88, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/d4b120326aae42afa663ee40c6161fa1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1444aadcb0664686b1e79c0463f72391, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/276fb6d14a4040c08f82706656eaa11e] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=160.2 K 2023-04-22 19:55:41,468 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/0c45922c7bf149668a371ff167d988cc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1b7fb611597b48ef89279a38ffe56110, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/d8d2f63e5aef41d599cbb4684b8479cb, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7c2d2c68a2b8472bb095fb65107ebc90, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/9a88bc6c636242b082cdf8e4d6131e74, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2c2f2dca306e49d696ebd08635334d80] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=69.7 K 2023-04-22 19:55:41,469 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 88ec84e2b4c14b628f18dce9633687a5, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682193332884 2023-04-22 19:55:41,469 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0c45922c7bf149668a371ff167d988cc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682193332884 2023-04-22 19:55:41,469 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting aecce56f222746a49a614bfd1a5a30a2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1682193333607 2023-04-22 19:55:41,469 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1b7fb611597b48ef89279a38ffe56110, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1682193333607 2023-04-22 19:55:41,469 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 70867959e8d84f95a7641f9c067f5e88, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682193335818 2023-04-22 19:55:41,469 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting d8d2f63e5aef41d599cbb4684b8479cb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682193335935 2023-04-22 19:55:41,469 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting d4b120326aae42afa663ee40c6161fa1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682193336430 2023-04-22 19:55:41,470 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 7c2d2c68a2b8472bb095fb65107ebc90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682193336430 2023-04-22 19:55:41,470 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 1444aadcb0664686b1e79c0463f72391, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682193337573 2023-04-22 19:55:41,470 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 9a88bc6c636242b082cdf8e4d6131e74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682193337573 2023-04-22 19:55:41,470 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 276fb6d14a4040c08f82706656eaa11e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1682193339764 2023-04-22 19:55:41,470 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 2c2f2dca306e49d696ebd08635334d80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1682193339780 2023-04-22 19:55:41,479 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#B#compaction#374 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:41,486 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#A#compaction#375 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:41,892 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/fc982b029ccc4455a8b11eebc59e5e57 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/fc982b029ccc4455a8b11eebc59e5e57 2023-04-22 19:55:41,896 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/B of 75f361e1b049d8d7dacb8de0d0756852 into fc982b029ccc4455a8b11eebc59e5e57(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:41,896 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:41,896 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/B, priority=10, startTime=1682193341466; duration=0sec 2023-04-22 19:55:41,897 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:41,897 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:55:41,898 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59202 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:55:41,898 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 75f361e1b049d8d7dacb8de0d0756852/C is initiating minor compaction (all files) 2023-04-22 19:55:41,898 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75f361e1b049d8d7dacb8de0d0756852/C in TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:41,898 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/bef44c9b53b14293ad17b1b6555a8802, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9dc376fbce7847bebc0db0190e3cf2f4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a9f2988d9c9741b78ba92f401e7ca8e7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/af1b1dd4193847849ad6e063ae272b43, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/d4bda3df63224615b4bfb9ee5ff932ce] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp, totalSize=57.8 K 2023-04-22 19:55:41,898 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting bef44c9b53b14293ad17b1b6555a8802, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1682193333607 2023-04-22 19:55:41,898 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 9dc376fbce7847bebc0db0190e3cf2f4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682193335935 2023-04-22 19:55:41,899 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a9f2988d9c9741b78ba92f401e7ca8e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682193336430 2023-04-22 19:55:41,899 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting af1b1dd4193847849ad6e063ae272b43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682193337573 2023-04-22 19:55:41,899 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting d4bda3df63224615b4bfb9ee5ff932ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1682193339780 2023-04-22 19:55:41,903 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/daf8de68555d4066a33782359a675d8b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/daf8de68555d4066a33782359a675d8b 2023-04-22 19:55:41,909 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 75f361e1b049d8d7dacb8de0d0756852#C#compaction#376 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:41,919 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/A of 75f361e1b049d8d7dacb8de0d0756852 into daf8de68555d4066a33782359a675d8b(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:41,919 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:41,919 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/A, priority=10, startTime=1682193341466; duration=0sec 2023-04-22 19:55:41,919 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:41,925 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/7331541cae9846069fff8bb56c7bcf10 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/7331541cae9846069fff8bb56c7bcf10 2023-04-22 19:55:41,928 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 75f361e1b049d8d7dacb8de0d0756852/C of 75f361e1b049d8d7dacb8de0d0756852 into 7331541cae9846069fff8bb56c7bcf10(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:41,928 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:41,929 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852., storeName=75f361e1b049d8d7dacb8de0d0756852/C, priority=11, startTime=1682193341466; duration=0sec 2023-04-22 19:55:41,929 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:41,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:41,995 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-22 19:55:41,995 DEBUG [Thread-2244] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bb27fd4 to 127.0.0.1:53381 2023-04-22 19:55:41,995 DEBUG [Thread-2244] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:41,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:41,997 DEBUG [Thread-2248] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d010713 to 127.0.0.1:53381 2023-04-22 19:55:41,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:41,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:41,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:41,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:41,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:41,997 DEBUG [Thread-2248] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:41,997 DEBUG [Thread-2242] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6434ce29 to 127.0.0.1:53381 2023-04-22 19:55:41,998 DEBUG [Thread-2242] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:42,005 DEBUG [Thread-2246] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x445919fd to 127.0.0.1:53381 2023-04-22 19:55:42,005 DEBUG [Thread-2246] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:42,007 DEBUG [Thread-2240] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b77d6d7 to 127.0.0.1:53381 2023-04-22 19:55:42,007 DEBUG [Thread-2240] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:42,007 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 19:55:42,007 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-22 19:55:42,007 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 230 2023-04-22 19:55:42,007 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-04-22 19:55:42,007 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-04-22 19:55:42,007 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1620 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 4855 rows 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1613 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 4833 rows 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1620 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 4857 rows 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1605 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 4815 rows 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1620 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 4856 rows 2023-04-22 19:55:42,008 INFO [Listener at localhost/46665] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 19:55:42,008 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x234f829f to 127.0.0.1:53381 2023-04-22 19:55:42,008 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:55:42,011 INFO [Listener at localhost/46665] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 19:55:42,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 19:55:42,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:42,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 19:55:42,020 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193342020"}]},"ts":"1682193342020"} 2023-04-22 19:55:42,021 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 19:55:42,022 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422f3156a341cac44c4b4ae03740b1d1e1a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f3156a341cac44c4b4ae03740b1d1e1a 2023-04-22 19:55:42,027 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=527, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/45ae534a9640465c88007924385436df 2023-04-22 19:55:42,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 19:55:42,206 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 19:55:42,207 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=75f361e1b049d8d7dacb8de0d0756852, UNASSIGN}] 2023-04-22 19:55:42,208 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=75f361e1b049d8d7dacb8de0d0756852, UNASSIGN 2023-04-22 19:55:42,209 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=75f361e1b049d8d7dacb8de0d0756852, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:42,209 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193342209"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193342209"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193342209"}]},"ts":"1682193342209"} 2023-04-22 19:55:42,210 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:55:42,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 19:55:42,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:42,362 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 75f361e1b049d8d7dacb8de0d0756852, disabling compactions & flushes 2023-04-22 19:55:42,362 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:42,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/f886b522cd28435e8039fa02b5b90b1d 2023-04-22 19:55:42,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 19:55:42,849 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/c632732c5b8745b49953a92acc254a28 2023-04-22 19:55:42,854 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/45ae534a9640465c88007924385436df as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/45ae534a9640465c88007924385436df 2023-04-22 19:55:42,857 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/45ae534a9640465c88007924385436df, entries=150, sequenceid=527, filesize=25.4 K 2023-04-22 19:55:42,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/f886b522cd28435e8039fa02b5b90b1d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/f886b522cd28435e8039fa02b5b90b1d 2023-04-22 19:55:42,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/f886b522cd28435e8039fa02b5b90b1d, entries=150, sequenceid=527, filesize=11.9 K 2023-04-22 19:55:42,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/c632732c5b8745b49953a92acc254a28 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c632732c5b8745b49953a92acc254a28 2023-04-22 19:55:42,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c632732c5b8745b49953a92acc254a28, entries=150, sequenceid=527, filesize=11.9 K 2023-04-22 19:55:42,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=13.42 KB/13740 for 75f361e1b049d8d7dacb8de0d0756852 in 871ms, sequenceid=527, compaction requested=false 2023-04-22 19:55:42,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:42,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:42,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:42,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. after waiting 0 ms 2023-04-22 19:55:42,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:42,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 75f361e1b049d8d7dacb8de0d0756852 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-22 19:55:42,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=A 2023-04-22 19:55:42,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:42,867 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=B 2023-04-22 19:55:42,867 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:42,867 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75f361e1b049d8d7dacb8de0d0756852, store=C 2023-04-22 19:55:42,867 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:43,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 19:55:43,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422b825c95f9bfd49be88c94a48a7109a89 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b825c95f9bfd49be88c94a48a7109a89 2023-04-22 19:55:43,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=532, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/8f40419aa3f94566afbc7ee15e11ff27 2023-04-22 19:55:44,094 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/1897e9dfc40c43019d54188fb9083ee2 2023-04-22 19:55:44,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 19:55:44,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/c46afab7ec06494698b783d9ae492122 2023-04-22 19:55:44,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/A/8f40419aa3f94566afbc7ee15e11ff27 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/8f40419aa3f94566afbc7ee15e11ff27 2023-04-22 19:55:44,512 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/8f40419aa3f94566afbc7ee15e11ff27, entries=100, sequenceid=532, filesize=18.6 K 2023-04-22 19:55:44,512 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/B/1897e9dfc40c43019d54188fb9083ee2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1897e9dfc40c43019d54188fb9083ee2 2023-04-22 19:55:44,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1897e9dfc40c43019d54188fb9083ee2, entries=100, sequenceid=532, filesize=9.5 K 2023-04-22 19:55:44,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/.tmp/C/c46afab7ec06494698b783d9ae492122 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c46afab7ec06494698b783d9ae492122 2023-04-22 19:55:44,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c46afab7ec06494698b783d9ae492122, entries=100, sequenceid=532, filesize=9.5 K 2023-04-22 19:55:44,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 75f361e1b049d8d7dacb8de0d0756852 in 1662ms, sequenceid=532, compaction requested=true 2023-04-22 19:55:44,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/88ec84e2b4c14b628f18dce9633687a5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/aecce56f222746a49a614bfd1a5a30a2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/70867959e8d84f95a7641f9c067f5e88, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/d4b120326aae42afa663ee40c6161fa1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1444aadcb0664686b1e79c0463f72391, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/276fb6d14a4040c08f82706656eaa11e] to archive 2023-04-22 19:55:44,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:55:44,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/88ec84e2b4c14b628f18dce9633687a5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/88ec84e2b4c14b628f18dce9633687a5 2023-04-22 19:55:44,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/aecce56f222746a49a614bfd1a5a30a2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/aecce56f222746a49a614bfd1a5a30a2 2023-04-22 19:55:44,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/70867959e8d84f95a7641f9c067f5e88 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/70867959e8d84f95a7641f9c067f5e88 2023-04-22 19:55:44,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/d4b120326aae42afa663ee40c6161fa1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/d4b120326aae42afa663ee40c6161fa1 2023-04-22 19:55:44,534 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1444aadcb0664686b1e79c0463f72391 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/1444aadcb0664686b1e79c0463f72391 2023-04-22 19:55:44,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/276fb6d14a4040c08f82706656eaa11e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/276fb6d14a4040c08f82706656eaa11e 2023-04-22 19:55:44,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/0c45922c7bf149668a371ff167d988cc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1b7fb611597b48ef89279a38ffe56110, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/d8d2f63e5aef41d599cbb4684b8479cb, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7c2d2c68a2b8472bb095fb65107ebc90, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/9a88bc6c636242b082cdf8e4d6131e74, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2c2f2dca306e49d696ebd08635334d80] to archive 2023-04-22 19:55:44,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:55:44,537 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/0c45922c7bf149668a371ff167d988cc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/0c45922c7bf149668a371ff167d988cc 2023-04-22 19:55:44,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1b7fb611597b48ef89279a38ffe56110 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1b7fb611597b48ef89279a38ffe56110 2023-04-22 19:55:44,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/d8d2f63e5aef41d599cbb4684b8479cb to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/d8d2f63e5aef41d599cbb4684b8479cb 2023-04-22 19:55:44,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7c2d2c68a2b8472bb095fb65107ebc90 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/7c2d2c68a2b8472bb095fb65107ebc90 2023-04-22 19:55:44,542 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/9a88bc6c636242b082cdf8e4d6131e74 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/9a88bc6c636242b082cdf8e4d6131e74 2023-04-22 19:55:44,543 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2c2f2dca306e49d696ebd08635334d80 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/2c2f2dca306e49d696ebd08635334d80 2023-04-22 19:55:44,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/bef44c9b53b14293ad17b1b6555a8802, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9dc376fbce7847bebc0db0190e3cf2f4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a9f2988d9c9741b78ba92f401e7ca8e7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/af1b1dd4193847849ad6e063ae272b43, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/d4bda3df63224615b4bfb9ee5ff932ce] to archive 2023-04-22 19:55:44,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:55:44,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/bef44c9b53b14293ad17b1b6555a8802 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/bef44c9b53b14293ad17b1b6555a8802 2023-04-22 19:55:44,547 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9dc376fbce7847bebc0db0190e3cf2f4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/9dc376fbce7847bebc0db0190e3cf2f4 2023-04-22 19:55:44,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a9f2988d9c9741b78ba92f401e7ca8e7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/a9f2988d9c9741b78ba92f401e7ca8e7 2023-04-22 19:55:44,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/af1b1dd4193847849ad6e063ae272b43 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/af1b1dd4193847849ad6e063ae272b43 2023-04-22 19:55:44,550 DEBUG [StoreCloser-TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/d4bda3df63224615b4bfb9ee5ff932ce to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/d4bda3df63224615b4bfb9ee5ff932ce 2023-04-22 19:55:44,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/recovered.edits/535.seqid, newMaxSeqId=535, maxSeqId=4 2023-04-22 19:55:44,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852. 2023-04-22 19:55:44,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 75f361e1b049d8d7dacb8de0d0756852: 2023-04-22 19:55:44,556 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:44,556 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=75f361e1b049d8d7dacb8de0d0756852, regionState=CLOSED 2023-04-22 19:55:44,557 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193344556"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193344556"}]},"ts":"1682193344556"} 2023-04-22 19:55:44,558 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-22 19:55:44,559 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 75f361e1b049d8d7dacb8de0d0756852, server=jenkins-hbase4.apache.org,44791,1682193215766 in 2.3470 sec 2023-04-22 19:55:44,560 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-22 19:55:44,560 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=75f361e1b049d8d7dacb8de0d0756852, UNASSIGN in 2.3520 sec 2023-04-22 19:55:44,561 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193344561"}]},"ts":"1682193344561"} 2023-04-22 19:55:44,562 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 19:55:44,563 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 19:55:44,565 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.5520 sec 2023-04-22 19:55:46,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 19:55:46,126 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-22 19:55:46,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 19:55:46,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:46,129 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:46,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-22 19:55:46,129 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:46,134 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:46,137 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/recovered.edits] 2023-04-22 19:55:46,140 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/45ae534a9640465c88007924385436df to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/45ae534a9640465c88007924385436df 2023-04-22 19:55:46,141 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/8f40419aa3f94566afbc7ee15e11ff27 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/8f40419aa3f94566afbc7ee15e11ff27 2023-04-22 19:55:46,143 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/daf8de68555d4066a33782359a675d8b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/A/daf8de68555d4066a33782359a675d8b 2023-04-22 19:55:46,145 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1897e9dfc40c43019d54188fb9083ee2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/1897e9dfc40c43019d54188fb9083ee2 2023-04-22 19:55:46,147 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/f886b522cd28435e8039fa02b5b90b1d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/f886b522cd28435e8039fa02b5b90b1d 2023-04-22 19:55:46,148 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/fc982b029ccc4455a8b11eebc59e5e57 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/B/fc982b029ccc4455a8b11eebc59e5e57 2023-04-22 19:55:46,150 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/7331541cae9846069fff8bb56c7bcf10 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/7331541cae9846069fff8bb56c7bcf10 2023-04-22 19:55:46,152 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c46afab7ec06494698b783d9ae492122 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c46afab7ec06494698b783d9ae492122 2023-04-22 19:55:46,153 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c632732c5b8745b49953a92acc254a28 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/C/c632732c5b8745b49953a92acc254a28 2023-04-22 19:55:46,155 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/recovered.edits/535.seqid to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852/recovered.edits/535.seqid 2023-04-22 19:55:46,156 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/75f361e1b049d8d7dacb8de0d0756852 2023-04-22 19:55:46,156 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:55:46,156 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 19:55:46,157 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-22 19:55:46,159 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220399ed306c884f42855f333789a56956 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220399ed306c884f42855f333789a56956 2023-04-22 19:55:46,161 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042208cae03aaa32496f9cee9f0e114620de to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042208cae03aaa32496f9cee9f0e114620de 2023-04-22 19:55:46,162 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422095bf7754b1f408b81f138e32aa63e7c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422095bf7754b1f408b81f138e32aa63e7c 2023-04-22 19:55:46,163 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304222e312cabfee7442e918c97113badb8d8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304222e312cabfee7442e918c97113badb8d8 2023-04-22 19:55:46,164 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042231a2e21e71554e068fb1425cfffdd7c3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042231a2e21e71554e068fb1425cfffdd7c3 2023-04-22 19:55:46,165 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422330ae0975c0a44949cf3cd32f31e3f17 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422330ae0975c0a44949cf3cd32f31e3f17 2023-04-22 19:55:46,166 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042241b2fed999d74e44be2a8f824f80f865 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042241b2fed999d74e44be2a8f824f80f865 2023-04-22 19:55:46,167 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042243bf331174714628a497a1e23d3d1391 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042243bf331174714628a497a1e23d3d1391 2023-04-22 19:55:46,168 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224ccd626995264ea7bc268420fedd54c4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224ccd626995264ea7bc268420fedd54c4 2023-04-22 19:55:46,169 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225abf77375d254f58805ac5477941060e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225abf77375d254f58805ac5477941060e 2023-04-22 19:55:46,170 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042264d7e939ebb748869574edc7e75f33ba to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042264d7e939ebb748869574edc7e75f33ba 2023-04-22 19:55:46,171 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304227f3769bb13ed4e698a00762fb18f9074 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304227f3769bb13ed4e698a00762fb18f9074 2023-04-22 19:55:46,172 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a3dc1ddb709943189b5fc085e89675c8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a3dc1ddb709943189b5fc085e89675c8 2023-04-22 19:55:46,173 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a632efa07ece4a2781ada8989e7f2b97 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a632efa07ece4a2781ada8989e7f2b97 2023-04-22 19:55:46,174 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a6f28ae6e4ff43818e2d3a0b27acbfc0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a6f28ae6e4ff43818e2d3a0b27acbfc0 2023-04-22 19:55:46,175 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b3650c7beef749bab68408bc6e0492b4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b3650c7beef749bab68408bc6e0492b4 2023-04-22 19:55:46,176 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b825c95f9bfd49be88c94a48a7109a89 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b825c95f9bfd49be88c94a48a7109a89 2023-04-22 19:55:46,177 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c3ebef4088ba42c2b0e7d6a38ace6c92 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c3ebef4088ba42c2b0e7d6a38ace6c92 2023-04-22 19:55:46,178 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ce23013ce4534b8685949503c2a5b455 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ce23013ce4534b8685949503c2a5b455 2023-04-22 19:55:46,179 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cea9a8b3213042fa9cace314fb572755 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cea9a8b3213042fa9cace314fb572755 2023-04-22 19:55:46,180 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d4df8b92b57c4f83aa4bd467826a6811 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d4df8b92b57c4f83aa4bd467826a6811 2023-04-22 19:55:46,181 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e9c6ab83abea41ef99c4cf57a2df8f23 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e9c6ab83abea41ef99c4cf57a2df8f23 2023-04-22 19:55:46,182 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f3156a341cac44c4b4ae03740b1d1e1a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f3156a341cac44c4b4ae03740b1d1e1a 2023-04-22 19:55:46,183 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f387525c6b38439880fc3c6096e5b5fa to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f387525c6b38439880fc3c6096e5b5fa 2023-04-22 19:55:46,183 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 19:55:46,185 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:46,186 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 19:55:46,188 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 19:55:46,189 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:46,189 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 19:55:46,189 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682193346189"}]},"ts":"9223372036854775807"} 2023-04-22 19:55:46,191 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 19:55:46,191 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 75f361e1b049d8d7dacb8de0d0756852, NAME => 'TestAcidGuarantees,,1682193317010.75f361e1b049d8d7dacb8de0d0756852.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 19:55:46,191 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 19:55:46,191 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682193346191"}]},"ts":"9223372036854775807"} 2023-04-22 19:55:46,192 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 19:55:46,194 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:46,195 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 68 msec 2023-04-22 19:55:46,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-22 19:55:46,230 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-22 19:55:46,242 INFO [Listener at localhost/46665] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=230 (was 224) - Thread LEAK? -, OpenFileDescriptor=542 (was 527) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=413 (was 291) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12727 (was 12507) - AvailableMemoryMB LEAK? - 2023-04-22 19:55:46,254 INFO [Listener at localhost/46665] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=230, OpenFileDescriptor=542, MaxFileDescriptor=60000, SystemLoadAverage=413, ProcessCount=170, AvailableMemoryMB=12727 2023-04-22 19:55:46,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 19:55:46,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 19:55:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 19:55:46,260 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 19:55:46,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-22 19:55:46,260 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 19:55:46,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 19:55:46,261 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:46,262 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1 empty. 2023-04-22 19:55:46,262 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:46,262 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:55:46,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 19:55:46,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 19:55:46,669 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 19:55:46,671 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 62116471b8dabdbd8f57094f112778d1, NAME => 'TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp 2023-04-22 19:55:46,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 19:55:47,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:55:47,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 62116471b8dabdbd8f57094f112778d1, disabling compactions & flushes 2023-04-22 19:55:47,077 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:47,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:47,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. after waiting 0 ms 2023-04-22 19:55:47,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:47,077 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:47,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:47,079 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 19:55:47,080 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193347080"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193347080"}]},"ts":"1682193347080"} 2023-04-22 19:55:47,081 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 19:55:47,082 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 19:55:47,082 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193347082"}]},"ts":"1682193347082"} 2023-04-22 19:55:47,082 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 19:55:47,086 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=62116471b8dabdbd8f57094f112778d1, ASSIGN}] 2023-04-22 19:55:47,088 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=62116471b8dabdbd8f57094f112778d1, ASSIGN 2023-04-22 19:55:47,089 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=62116471b8dabdbd8f57094f112778d1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44791,1682193215766; forceNewPlan=false, retain=false 2023-04-22 19:55:47,240 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=62116471b8dabdbd8f57094f112778d1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:47,240 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193347240"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193347240"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193347240"}]},"ts":"1682193347240"} 2023-04-22 19:55:47,242 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:55:47,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 19:55:47,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:47,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 62116471b8dabdbd8f57094f112778d1, NAME => 'TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:55:47,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:47,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:55:47,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:47,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:47,398 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:47,399 DEBUG [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A 2023-04-22 19:55:47,399 DEBUG [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A 2023-04-22 19:55:47,400 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:55:47,400 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 62116471b8dabdbd8f57094f112778d1 columnFamilyName A 2023-04-22 19:55:47,400 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] regionserver.HStore(310): Store=62116471b8dabdbd8f57094f112778d1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:55:47,400 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:47,401 DEBUG [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B 2023-04-22 19:55:47,401 DEBUG [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B 2023-04-22 19:55:47,402 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:55:47,402 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 62116471b8dabdbd8f57094f112778d1 columnFamilyName B 2023-04-22 19:55:47,402 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] regionserver.HStore(310): Store=62116471b8dabdbd8f57094f112778d1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:55:47,402 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:47,403 DEBUG [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C 2023-04-22 19:55:47,403 DEBUG [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C 2023-04-22 19:55:47,403 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:55:47,403 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 62116471b8dabdbd8f57094f112778d1 columnFamilyName C 2023-04-22 19:55:47,404 INFO [StoreOpener-62116471b8dabdbd8f57094f112778d1-1] regionserver.HStore(310): Store=62116471b8dabdbd8f57094f112778d1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:55:47,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:47,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:47,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:47,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 19:55:47,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:47,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 19:55:47,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 62116471b8dabdbd8f57094f112778d1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66800319, jitterRate=-0.004597678780555725}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 19:55:47,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:47,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., pid=49, masterSystemTime=1682193347393 2023-04-22 19:55:47,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:47,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:47,412 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=62116471b8dabdbd8f57094f112778d1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:47,412 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193347412"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682193347412"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682193347412"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682193347412"}]},"ts":"1682193347412"} 2023-04-22 19:55:47,414 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-22 19:55:47,414 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 in 171 msec 2023-04-22 19:55:47,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-22 19:55:47,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=62116471b8dabdbd8f57094f112778d1, ASSIGN in 328 msec 2023-04-22 19:55:47,416 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 19:55:47,416 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193347416"}]},"ts":"1682193347416"} 2023-04-22 19:55:47,416 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 19:55:47,418 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 19:55:47,419 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-22 19:55:48,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 19:55:48,366 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-22 19:55:48,366 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x3a29ce5a to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:48,371 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73eb7080, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:48,372 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:48,375 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:48,376 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 19:55:48,377 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39950, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 19:55:48,379 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x5e372785 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:48,382 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a987907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:48,383 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x501c5a88 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:48,386 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37705401, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:48,387 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x27f2d997 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:48,392 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15880ffb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:48,392 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x34ad4aa8 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:48,397 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52203cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:48,397 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x1c29757d to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:48,402 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420f945a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:48,402 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:48,407 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:48,407 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:48,413 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:48,413 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:48,418 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:48,418 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x5f5363d8 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:48,422 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78592df0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:48,422 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x211582f2 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:55:48,427 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@659c68ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:55:48,437 DEBUG [hconnection-0x51802553-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:48,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:48,443 DEBUG [hconnection-0x2f7b5370-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:48,443 DEBUG [hconnection-0x567bd078-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:48,444 DEBUG [hconnection-0x6e1d00f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:48,444 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40126, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:48,452 DEBUG [hconnection-0x76be57cc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:48,452 DEBUG [hconnection-0x54cfef37-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:48,452 DEBUG [hconnection-0x3d907a5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:48,452 DEBUG [hconnection-0xb7dd009-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:48,452 DEBUG [hconnection-0x64ed056b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:48,452 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40116, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:48,453 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:48,453 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40140, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:48,453 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40174, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:48,454 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40134, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:48,455 DEBUG [hconnection-0x300a9c3d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:55:48,463 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40156, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:48,464 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40142, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:48,464 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40170, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:48,464 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:48,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:48,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:48,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:48,465 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,465 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,467 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:48,467 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:48,469 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40190, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:55:48,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:48,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:48,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:48,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:48,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:48,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:48,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:48,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:48,480 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 19:55:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:48,480 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:48,482 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:48,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:48,482 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:48,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:48,483 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:48,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:48,483 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:48,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:48,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,485 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:48,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@129c9142[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:48,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:48,487 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:48,489 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,489 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:48,490 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,491 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:48,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:48,494 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:48,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:48,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:48,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:48,494 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:48,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,497 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,498 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193408497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193408498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193408499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193348494, max timestamp=1682193348499Num uniques 859; 2023-04-22 19:55:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193348494, max timestamp=1682193348499Num uniques 859; 2023-04-22 19:55:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:55:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:55:48,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193408499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193348494, max timestamp=1682193348499Num uniques 859; 2023-04-22 19:55:48,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:55:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,511 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:48,511 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,511 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:48,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:48,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:48,511 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,511 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:48,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:48,511 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:48,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:48,511 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,511 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:48,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:48,511 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,512 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,512 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,512 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40190 deadline: 1682193408532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193408602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193408602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193408602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193408602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,611 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:48,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:48,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:48,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:48,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:48,616 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:48,616 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:48,616 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:48,616 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:48,616 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,616 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,617 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,617 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:48,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:48,618 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,618 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:48,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:48,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:48,631 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:48,631 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:48,631 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:48,631 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:48,631 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:48,631 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:48,631 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:48,631 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,631 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:48,631 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:48,632 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,633 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,633 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:48,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b4d7c12[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:48,634 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:48,634 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,635 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:48,636 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,636 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:48,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40190 deadline: 1682193408633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,638 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:48,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,638 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:48,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:48,641 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:48,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:48,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:48,641 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:48,641 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:48,642 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,642 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,642 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:48,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,644 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,645 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:48,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:48,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:48,647 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,647 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:48,647 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:48,647 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,647 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:48,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:48,647 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:48,647 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:48,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:48,747 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:48,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:48,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:48,756 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:48,756 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:48,756 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:48,756 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:48,756 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,757 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,758 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:48,758 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:48,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:48,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:48,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:48,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:48,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:48,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:48,766 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:48,766 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:48,766 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:48,766 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:48,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,769 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,770 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,770 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:48,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8f6ede[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:48,771 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:48,771 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,772 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:48,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:48,772 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,772 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:48,772 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:48,773 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:48,775 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:48,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:48,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:48,775 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:48,775 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:48,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,778 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,778 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:48,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:48,778 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,779 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,782 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:48,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:48,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:48,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:48,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:48,783 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:48,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:48,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:48,783 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:48,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,783 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:48,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193408810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193408811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193408811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193408811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:48,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40190 deadline: 1682193408840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,883 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:48,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:48,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:48,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:48,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:48,887 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:48,887 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:48,887 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:48,887 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:48,888 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,888 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:48,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:48,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,889 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:48,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:48,890 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:48,890 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:48,890 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:48,890 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:48,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:48,891 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:48,891 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:48,891 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:48,891 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:48,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:48,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:48,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:48,891 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:48,892 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,892 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,893 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:48,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@484feac[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:48,894 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:48,894 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,895 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:48,895 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,895 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:48,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:48,895 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:48,898 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:48,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:48,900 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:48,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:48,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:48,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:48,900 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:48,902 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:48,902 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,902 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:48,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:48,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,904 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:48,909 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,914 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:48,914 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:48,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:48,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:48,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:48,914 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,915 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:48,915 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:48,915 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,915 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:48,914 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:48,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:48,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:48,915 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:48,915 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:48,915 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:48,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/9660a007f5644680a1be7a919b852261 2023-04-22 19:55:49,015 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:49,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:49,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:49,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:49,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:49,019 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:49,019 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,019 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:49,019 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:49,020 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,020 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,022 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,022 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,023 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:49,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:49,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:49,025 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,025 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:49,025 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:49,025 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:49,026 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,027 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,027 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,027 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:49,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72e479ff[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:49,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:49,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:49,031 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:49,032 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,032 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,032 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,032 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:49,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:49,034 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:49,034 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:49,034 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:49,035 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:49,035 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:49,037 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,037 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,037 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,037 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,039 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:49,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:49,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,041 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,041 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:49,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,041 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:49,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,041 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:49,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193409117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:49,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193409118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:49,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193409118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:49,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193409119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,141 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:49,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:49,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:49,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:49,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:49,145 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:49,145 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,146 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:49,146 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:49,146 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,146 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,147 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,147 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,147 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,148 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:49,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40190 deadline: 1682193409147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:49,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:49,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:49,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:49,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:49,149 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,149 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:49,149 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:49,150 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:49,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:49,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:49,150 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:49,151 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52e9265d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:49,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:49,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,154 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:49,155 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,155 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:49,156 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,156 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:49,156 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,156 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:49,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:49,158 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:49,158 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:49,158 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:49,158 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:49,158 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:49,160 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,160 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,160 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,161 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,162 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,164 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,164 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,164 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,164 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,164 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:49,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:49,164 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,165 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,165 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,165 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,165 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:49,165 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:49,165 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:49,265 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:49,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:49,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:49,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:49,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:49,269 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:49,269 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,269 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:49,269 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:49,269 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,269 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,271 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,271 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,272 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,272 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:49,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:49,273 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,273 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:49,273 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:49,273 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:49,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,275 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:49,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c44b40[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:49,277 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:49,277 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,278 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:49,278 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,278 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:49,279 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,279 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,279 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,279 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:49,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:49,281 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:49,282 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:49,282 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:49,282 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:49,282 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:49,283 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,284 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,284 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,284 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:49,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:49,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,289 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,290 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:49,290 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/5e019f4a0a6f44cbb3c9952cda27e701 2023-04-22 19:55:49,390 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:49,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:49,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:49,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:49,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:49,394 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:49,394 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,394 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:49,394 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:49,396 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,396 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,401 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,401 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,402 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,402 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:49,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:49,404 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:49,404 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,404 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:49,405 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,405 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:49,406 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:49,406 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,412 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:49,414 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:49,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2516b4aa[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:49,414 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,415 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,415 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:49,415 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:49,415 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:49,415 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,415 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:49,417 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,417 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,417 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:49,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:49,421 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:49,421 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:49,421 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:49,421 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:49,421 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:49,422 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,423 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,424 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,424 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,425 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,429 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,429 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,429 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,429 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,429 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,429 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,430 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:49,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:49,430 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:49,431 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:49,531 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:49,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:49,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:49,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:49,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:49,535 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:49,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:49,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:49,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,535 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,536 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,536 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,537 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,537 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:49,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:49,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:49,538 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:49,538 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,538 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:49,538 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:49,538 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:49,538 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,539 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:49,539 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:49,539 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,539 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:49,539 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:49,539 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,539 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:49,539 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:49,540 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,540 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,541 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,541 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:49,542 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:49,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74ec1a76[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:49,543 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,543 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,544 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:49,544 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:49,544 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:49,544 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,544 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:49,546 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,547 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:49,547 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,547 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:49,549 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:49,549 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:49,549 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:49,549 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:49,549 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:49,551 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,551 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,551 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,552 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,553 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,556 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,556 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,556 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,556 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,556 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,556 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,556 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,556 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,556 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:49,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:49,556 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,557 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:49,557 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:49,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:49,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193409623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:49,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193409624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:49,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193409625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:49,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193409625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:49,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40190 deadline: 1682193409653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,657 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:49,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:49,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:49,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:49,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:49,660 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:49,660 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,660 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:49,660 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:49,661 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,661 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,662 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,662 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,662 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,662 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:49,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:49,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:49,663 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,663 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:49,664 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:49,664 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:49,666 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,666 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,666 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,666 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:49,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f2c1178[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:49,667 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:49,667 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,668 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,668 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:49,668 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:49,668 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:49,669 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,669 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:49,670 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,670 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,670 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,670 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:49,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:49,672 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:49,673 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:49,673 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:49,673 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:49,673 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:49,674 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,674 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,674 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,674 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,675 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,679 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:49,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:49,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:49,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,679 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:49,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,680 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:49,680 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,680 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,680 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,680 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/ef7d9ab3d9e3405fa4fa94d244ac41db 2023-04-22 19:55:49,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/9660a007f5644680a1be7a919b852261 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9660a007f5644680a1be7a919b852261 2023-04-22 19:55:49,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9660a007f5644680a1be7a919b852261, entries=150, sequenceid=12, filesize=11.6 K 2023-04-22 19:55:49,780 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:49,780 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/5e019f4a0a6f44cbb3c9952cda27e701 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/5e019f4a0a6f44cbb3c9952cda27e701 2023-04-22 19:55:49,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:49,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:49,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:49,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:49,784 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:49,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:49,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:49,784 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,785 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,786 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,786 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,786 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:49,786 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/5e019f4a0a6f44cbb3c9952cda27e701, entries=150, sequenceid=12, filesize=11.6 K 2023-04-22 19:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:49,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:49,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:49,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:49,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/ef7d9ab3d9e3405fa4fa94d244ac41db as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ef7d9ab3d9e3405fa4fa94d244ac41db 2023-04-22 19:55:49,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:49,788 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,788 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:49,788 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:49,788 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:49,789 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:49,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ef7d9ab3d9e3405fa4fa94d244ac41db, entries=150, sequenceid=12, filesize=11.6 K 2023-04-22 19:55:49,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,793 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,793 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 62116471b8dabdbd8f57094f112778d1 in 1313ms, sequenceid=12, compaction requested=false 2023-04-22 19:55:49,793 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:49,793 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:49,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b16d996[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:49,794 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:49,794 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,796 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,796 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:49,797 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,797 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:49,797 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:49,797 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:49,798 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,798 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:49,798 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,799 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:49,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:49,801 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:49,801 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:49,801 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:49,801 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:49,801 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:49,805 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,805 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,805 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:49,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:49,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,806 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:49,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,808 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,810 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,810 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:49,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:49,811 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,811 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,811 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,811 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:49,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:49,811 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,811 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:49,811 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:49,811 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,812 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:49,812 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:49,912 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:49,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:49,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:49,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:49,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:49,915 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:49,915 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,915 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:49,915 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:49,916 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,916 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,921 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,921 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:49,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:49,921 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:49,921 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:49,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:49,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:49,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:49,924 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:49,924 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:49,924 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:49,924 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:49,924 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:49,924 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:49,924 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:49,924 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-22 19:55:49,924 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:49,924 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:49,924 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:49,924 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:49,925 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:49,925 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:50,086 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 19:55:50,334 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/93edf50fbce241c0b99c82b4c660ab12 2023-04-22 19:55:50,346 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/1ce6a0aa20f443819689b2dd92ff625c 2023-04-22 19:55:50,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:50,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:50,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:50,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193410645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:50,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193410646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:50,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193410647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:50,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193410652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:50,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40190 deadline: 1682193410655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:50,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193410753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:50,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193410753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,757 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/812604ed7d514be6a5a75259b298de1b 2023-04-22 19:55:50,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:50,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193410758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:50,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193410759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,761 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/93edf50fbce241c0b99c82b4c660ab12 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/93edf50fbce241c0b99c82b4c660ab12 2023-04-22 19:55:50,765 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/93edf50fbce241c0b99c82b4c660ab12, entries=150, sequenceid=46, filesize=11.6 K 2023-04-22 19:55:50,766 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/1ce6a0aa20f443819689b2dd92ff625c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1ce6a0aa20f443819689b2dd92ff625c 2023-04-22 19:55:50,772 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1ce6a0aa20f443819689b2dd92ff625c, entries=150, sequenceid=46, filesize=11.6 K 2023-04-22 19:55:50,772 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/812604ed7d514be6a5a75259b298de1b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/812604ed7d514be6a5a75259b298de1b 2023-04-22 19:55:50,776 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/812604ed7d514be6a5a75259b298de1b, entries=150, sequenceid=46, filesize=11.6 K 2023-04-22 19:55:50,777 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 62116471b8dabdbd8f57094f112778d1 in 852ms, sequenceid=46, compaction requested=false 2023-04-22 19:55:50,777 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:50,777 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:50,778 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:50,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:50,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b3587ba[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:50,780 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:50,780 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:50,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:50,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:50,781 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:50,781 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:50,781 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:50,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:50,783 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:50,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,783 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:50,783 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:50,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:50,785 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:50,785 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:50,785 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:50,785 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:50,785 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:50,787 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:50,787 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:50,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:50,788 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:50,788 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:50,789 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,791 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:50,791 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:50,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:50,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:50,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:50,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:50,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:50,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:50,792 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:50,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:50,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:50,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:50,792 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:50,792 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:50,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:50,893 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:50,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:50,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:50,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:50,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:50,896 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:50,896 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:50,896 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:50,896 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:50,896 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:50,896 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:50,897 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,897 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:50,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:50,898 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:50,898 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:50,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:50,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:50,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:50,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:50,899 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:50,899 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:50,899 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:50,899 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:50,900 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:50,900 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:50,900 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:50,900 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 19:55:50,900 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:50,900 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:50,900 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:50,900 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:50,900 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:50,900 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:50,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:50,958 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:50,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682193350972, max timestamp=1682193350985Num uniques 859; 2023-04-22 19:55:50,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-22 19:55:50,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682193350972, max timestamp=1682193350985Num uniques 859; 2023-04-22 19:55:50,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-22 19:55:50,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682193350972, max timestamp=1682193350985Num uniques 859; 2023-04-22 19:55:50,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-22 19:55:51,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193411011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193411011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193411012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193411013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193411117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193411118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193411120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193411120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,321 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/9236676d203c44f2a95bd2162e667fa9 2023-04-22 19:55:51,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193411322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193411322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193411326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193411327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,334 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/e257c1d05d6f4295bba16138e04e0051 2023-04-22 19:55:51,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193411626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193411627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193411634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:51,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193411636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,744 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/c8887bb66e714b3b9911c3771bdb2caf 2023-04-22 19:55:51,748 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/9236676d203c44f2a95bd2162e667fa9 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9236676d203c44f2a95bd2162e667fa9 2023-04-22 19:55:51,752 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9236676d203c44f2a95bd2162e667fa9, entries=150, sequenceid=57, filesize=11.6 K 2023-04-22 19:55:51,754 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/e257c1d05d6f4295bba16138e04e0051 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/e257c1d05d6f4295bba16138e04e0051 2023-04-22 19:55:51,758 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/e257c1d05d6f4295bba16138e04e0051, entries=150, sequenceid=57, filesize=11.6 K 2023-04-22 19:55:51,759 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/c8887bb66e714b3b9911c3771bdb2caf as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/c8887bb66e714b3b9911c3771bdb2caf 2023-04-22 19:55:51,762 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/c8887bb66e714b3b9911c3771bdb2caf, entries=150, sequenceid=57, filesize=11.6 K 2023-04-22 19:55:51,763 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 62116471b8dabdbd8f57094f112778d1 in 863ms, sequenceid=57, compaction requested=true 2023-04-22 19:55:51,763 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:51,763 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:51,764 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,764 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:51,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:51,765 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f9e0900[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:51,766 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:51,766 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:51,768 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:51,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:51,768 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,768 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:51,771 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:51,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:51,773 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:51,773 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:51,773 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:51,773 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:51,773 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:51,775 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:51,775 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:51,775 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:51,776 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:51,777 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:51,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:51,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:51,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:51,779 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:51,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:51,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:51,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:51,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:51,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,779 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:51,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:51,780 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:51,779 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:51,780 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:51,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:51,880 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:51,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:51,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:51,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:51,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:51,883 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:51,883 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:51,883 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:51,883 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:51,884 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:51,884 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:51,886 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:51,886 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:51,886 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:51,887 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:51,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:51,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:51,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:51,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:51,888 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:51,888 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:51,888 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-22 19:55:51,888 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:51,889 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:51,889 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:51,889 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:51,889 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:51,889 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:52,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:52,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:52,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193412151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193412151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193412152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193412158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193412261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193412261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193412262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193412267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,298 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/8846415461d44d1ba45ded2619946ea2 2023-04-22 19:55:52,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193412465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193412465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193412466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193412474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40190 deadline: 1682193412677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,685 DEBUG [Thread-2961] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:55:52,712 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/6aa4f8ef15d74e999eb061ea86038a20 2023-04-22 19:55:52,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193412769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193412770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193412770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:52,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:52,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193412779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,128 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/1f0a46d20a7444cab37eb691df7dbd93 2023-04-22 19:55:53,133 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/8846415461d44d1ba45ded2619946ea2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/8846415461d44d1ba45ded2619946ea2 2023-04-22 19:55:53,137 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/8846415461d44d1ba45ded2619946ea2, entries=150, sequenceid=91, filesize=11.6 K 2023-04-22 19:55:53,137 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/6aa4f8ef15d74e999eb061ea86038a20 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/6aa4f8ef15d74e999eb061ea86038a20 2023-04-22 19:55:53,141 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/6aa4f8ef15d74e999eb061ea86038a20, entries=150, sequenceid=91, filesize=11.6 K 2023-04-22 19:55:53,142 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/1f0a46d20a7444cab37eb691df7dbd93 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/1f0a46d20a7444cab37eb691df7dbd93 2023-04-22 19:55:53,146 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/1f0a46d20a7444cab37eb691df7dbd93, entries=150, sequenceid=91, filesize=11.6 K 2023-04-22 19:55:53,146 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 62116471b8dabdbd8f57094f112778d1 in 1258ms, sequenceid=91, compaction requested=true 2023-04-22 19:55:53,147 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:53,147 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:53,148 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,148 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40b6765a[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:53,150 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:53,150 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:53,151 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:53,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:53,151 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,152 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:53,156 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:53,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:53,161 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:53,161 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:53,161 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:53,161 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:53,161 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:53,163 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:53,163 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:53,163 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:53,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:53,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:53,164 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,166 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:53,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:53,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:53,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:53,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:53,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:53,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:53,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:53,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:53,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:53,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:53,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:53,170 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:53,170 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:53,271 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:53,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:53,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:53,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:53,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:53,275 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:53,275 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:53,275 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:53,275 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:53,275 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:53,275 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:53,277 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,277 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:53,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:53,277 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,277 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:53,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:53,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:53,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:53,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:53,278 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:53,278 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:53,278 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 19:55:53,278 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:53,279 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:53,279 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:53,279 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:53,279 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:53,279 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:53,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:53,281 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:53,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193413314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193413315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193353305, max timestamp=1682193353323Num uniques 859; 2023-04-22 19:55:53,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193353305, max timestamp=1682193353323Num uniques 859; 2023-04-22 19:55:53,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:55:53,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193353305, max timestamp=1682193353323Num uniques 859; 2023-04-22 19:55:53,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:55:53,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:55:53,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193413373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193413382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193413426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193413427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193413488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193413490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193413631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193413632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,690 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/2d5f56e2d6f34a549e77080cc845a7e6 2023-04-22 19:55:53,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193413695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,701 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/bd6b5abfac434e00bf08be959bf6f6d2 2023-04-22 19:55:53,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193413697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193413939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:53,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:53,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193413940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193413998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193414007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,112 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/d5bc1cfb6b4a4379a73411180f1c723c 2023-04-22 19:55:54,116 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/2d5f56e2d6f34a549e77080cc845a7e6 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2d5f56e2d6f34a549e77080cc845a7e6 2023-04-22 19:55:54,120 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2d5f56e2d6f34a549e77080cc845a7e6, entries=150, sequenceid=102, filesize=11.6 K 2023-04-22 19:55:54,121 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/bd6b5abfac434e00bf08be959bf6f6d2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bd6b5abfac434e00bf08be959bf6f6d2 2023-04-22 19:55:54,125 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bd6b5abfac434e00bf08be959bf6f6d2, entries=150, sequenceid=102, filesize=11.6 K 2023-04-22 19:55:54,125 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/d5bc1cfb6b4a4379a73411180f1c723c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d5bc1cfb6b4a4379a73411180f1c723c 2023-04-22 19:55:54,130 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d5bc1cfb6b4a4379a73411180f1c723c, entries=150, sequenceid=102, filesize=11.6 K 2023-04-22 19:55:54,130 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 62116471b8dabdbd8f57094f112778d1 in 852ms, sequenceid=102, compaction requested=true 2023-04-22 19:55:54,130 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:54,130 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:54,131 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:54,131 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,131 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:54,131 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:54,133 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,133 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,134 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,134 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d91ddb3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:54,135 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:54,135 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,136 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,137 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:54,137 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:54,137 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:54,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,137 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:54,138 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:54,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:54,139 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:54,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:54,141 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:54,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:54,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:54,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:54,141 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:54,142 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,142 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,142 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,142 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,144 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,150 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:54,150 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:54,150 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:54,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:54,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:54,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:54,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:54,150 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,151 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,151 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,151 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,151 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:54,151 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:54,151 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,151 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:54,151 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:54,251 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:54,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:54,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:54,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:54,255 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:54,255 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:54,255 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:54,255 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:54,256 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,256 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,257 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:54,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:54,257 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,257 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,257 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:54,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:54,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:54,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:54,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:54,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:54,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:54,260 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:54,260 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:54,260 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:54,260 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-22 19:55:54,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:54,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:54,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:54,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:54,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:54,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:54,275 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/2e472179723f4aa0803da70c4299466f 2023-04-22 19:55:54,286 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/bbbccc995cb541999e60f9b2a66f2116 2023-04-22 19:55:54,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:54,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:54,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193414471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193414480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193414511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193414513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193414584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193414589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,697 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/ebb268245b824ab6b3943a744ecdefa5 2023-04-22 19:55:54,702 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/2e472179723f4aa0803da70c4299466f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2e472179723f4aa0803da70c4299466f 2023-04-22 19:55:54,706 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2e472179723f4aa0803da70c4299466f, entries=150, sequenceid=136, filesize=11.8 K 2023-04-22 19:55:54,706 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/bbbccc995cb541999e60f9b2a66f2116 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bbbccc995cb541999e60f9b2a66f2116 2023-04-22 19:55:54,710 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bbbccc995cb541999e60f9b2a66f2116, entries=150, sequenceid=136, filesize=11.8 K 2023-04-22 19:55:54,711 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/ebb268245b824ab6b3943a744ecdefa5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ebb268245b824ab6b3943a744ecdefa5 2023-04-22 19:55:54,714 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ebb268245b824ab6b3943a744ecdefa5, entries=150, sequenceid=136, filesize=11.8 K 2023-04-22 19:55:54,715 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 62116471b8dabdbd8f57094f112778d1 in 455ms, sequenceid=136, compaction requested=true 2023-04-22 19:55:54,715 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:54,715 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:54,715 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:54,715 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:54,715 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,715 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:54,715 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:54,717 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,717 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,717 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,717 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:54,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aee85ec[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:54,719 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:54,719 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,720 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,720 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:54,720 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:54,720 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:54,720 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,720 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:54,723 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:54,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:54,728 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:54,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:54,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:54,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:54,729 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:54,730 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,730 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,730 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:54,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,731 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,734 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,737 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,738 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:54,738 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:54,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:54,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:54,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:54,738 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:54,738 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:54,738 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:54,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:54,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:54,797 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 19:55:54,797 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:54,797 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:54,797 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:54,797 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:54,797 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:54,797 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:54,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/03370b7bee704702b05777c9cf566bfc 2023-04-22 19:55:54,838 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:54,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:54,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:54,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:54,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:54,842 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:54,842 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:54,842 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:54,842 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:54,842 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,842 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,843 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:54,843 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:54,844 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,844 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:54,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:54,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:54,846 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:54,846 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:54,846 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:54,846 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:54,849 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,849 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,849 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,849 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:54,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@758753b8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:54,851 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:54,851 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,852 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:54,852 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,852 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:54,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:54,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:54,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:54,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:54,855 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:54,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:54,858 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:54,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:54,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:54,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:54,859 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:54,864 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:54,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,865 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,867 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:54,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:54,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:54,869 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:54,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:54,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:54,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:54,869 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:54,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,870 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:54,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:54,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193414866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193414868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,970 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:54,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:54,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:54,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:54,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:54,973 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:54,973 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:54,974 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:54,974 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:54,974 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,974 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,975 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:54,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:54,975 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,976 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,976 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:54,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:54,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:54,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:54,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:54,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:54,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:54,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:54,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:54,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:54,977 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:54,977 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:54,977 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:54,977 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:54,977 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:54,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:54,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:54,978 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:54,979 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,979 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,979 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,979 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:54,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f0eff4[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:54,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:54,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193414976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,983 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:54,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:54,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:54,983 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:54,983 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,983 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:54,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:54,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193414977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,985 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,985 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:54,985 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:54,985 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:54,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:54,987 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:54,987 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:54,987 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:54,987 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:54,987 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:54,989 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,989 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:54,989 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:54,989 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,990 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,001 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:55,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:55,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,002 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:55,002 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:55,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,102 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:55,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:55,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:55,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:55,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:55,106 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:55,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:55,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:55,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,109 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,109 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,111 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,111 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:55,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:55,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:55,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:55,112 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,112 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:55,113 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:55,113 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:55,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,114 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:55,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@353fc6dc[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:55,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:55,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,117 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,117 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:55,117 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:55,117 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:55,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,118 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:55,119 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:55,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,119 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:55,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:55,123 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:55,123 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:55,123 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:55,123 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:55,123 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:55,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,125 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,126 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,127 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,129 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,129 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,129 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,129 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,129 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,129 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,130 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,130 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,130 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:55,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:55,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,130 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,130 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:55,130 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:55,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:55,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193415185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:55,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193415185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/a8452519ef0d4efd8f60127a335417de 2023-04-22 19:55:55,230 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:55,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:55,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:55,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:55,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:55,234 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:55,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:55,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:55,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,235 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,235 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,237 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,237 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:55,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:55,239 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:55,239 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,239 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:55,239 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:55,239 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:55,239 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,239 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:55,240 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:55,240 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:55,241 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,241 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,241 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,242 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:55,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bc0e472[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:55,245 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:55,245 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,249 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,249 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:55,250 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:55,250 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:55,250 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,250 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:55,251 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:55,251 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,251 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,251 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:55,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:55,255 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:55,255 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:55,255 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:55,255 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:55,255 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:55,258 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,258 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,258 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,259 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,260 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,266 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,266 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,267 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,267 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:55,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:55,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,266 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,267 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,267 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:55,267 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:55,267 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,267 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,267 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,267 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,367 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:55,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:55,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:55,379 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:55,379 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,379 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:55,379 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:55,379 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,380 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,381 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,381 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,381 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,381 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:55,382 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,382 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:55,382 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:55,383 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,383 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:55,383 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:55,383 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,383 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:55,383 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,384 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:55,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44156c21[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:55,385 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:55,385 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,386 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,387 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:55,387 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:55,387 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:55,387 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,387 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:55,389 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:55,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:55,393 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:55,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:55,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:55,393 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:55,393 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:55,394 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,394 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,394 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,394 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,395 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:55,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:55,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,397 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,398 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,397 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,398 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,398 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:55,398 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,398 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,398 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:55,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:55,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,498 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:55,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:55,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:55,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:55,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193415491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:55,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:55,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:55,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193415492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,502 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:55,502 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,502 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:55,502 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:55,502 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,502 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,503 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,503 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,504 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,504 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:55,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:55,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:55,506 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,506 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:55,506 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:55,506 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,509 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,510 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:55,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@459bb418[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:55,511 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:55,511 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,512 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:55,513 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,513 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:55,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:55,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,514 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,514 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:55,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:55,518 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:55,518 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:55,518 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:55,518 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:55,518 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:55,519 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,519 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,519 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,520 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,521 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,524 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,524 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,524 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,524 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:55,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:55,525 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,525 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:55,525 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:55,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:55,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193415525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:55,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193415527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,625 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:55,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:55,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:55,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:55,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:55,629 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:55,629 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,629 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:55,629 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:55,629 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,630 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,631 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,631 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/67deee18b02e49769f9d9b2868fc62b5 2023-04-22 19:55:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:55,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:55,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:55,633 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:55,633 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:55,633 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,633 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:55,633 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:55,633 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,633 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:55,633 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:55,633 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,633 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:55,633 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:55,634 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,635 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:55,636 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/03370b7bee704702b05777c9cf566bfc as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/03370b7bee704702b05777c9cf566bfc 2023-04-22 19:55:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:55,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e598351[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:55,638 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:55,638 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,639 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,640 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:55,640 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:55,640 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:55,640 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,640 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:55,641 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/03370b7bee704702b05777c9cf566bfc, entries=200, sequenceid=148, filesize=14.1 K 2023-04-22 19:55:55,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/a8452519ef0d4efd8f60127a335417de as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/a8452519ef0d4efd8f60127a335417de 2023-04-22 19:55:55,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:55,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,642 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:55,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/a8452519ef0d4efd8f60127a335417de, entries=150, sequenceid=148, filesize=11.8 K 2023-04-22 19:55:55,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,646 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/67deee18b02e49769f9d9b2868fc62b5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/67deee18b02e49769f9d9b2868fc62b5 2023-04-22 19:55:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:55,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:55,649 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:55,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:55,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:55,649 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:55,649 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:55,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/67deee18b02e49769f9d9b2868fc62b5, entries=150, sequenceid=148, filesize=11.8 K 2023-04-22 19:55:55,650 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 62116471b8dabdbd8f57094f112778d1 in 854ms, sequenceid=148, compaction requested=true 2023-04-22 19:55:55,651 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:55,651 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:55,651 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:55,651 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 19:55:55,651 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 19:55:55,651 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:55,653 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85892 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 19:55:55,653 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83502 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 19:55:55,653 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/A is initiating minor compaction (all files) 2023-04-22 19:55:55,653 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/B is initiating minor compaction (all files) 2023-04-22 19:55:55,653 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/A in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,653 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/B in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,653 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9660a007f5644680a1be7a919b852261, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/93edf50fbce241c0b99c82b4c660ab12, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9236676d203c44f2a95bd2162e667fa9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/8846415461d44d1ba45ded2619946ea2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2d5f56e2d6f34a549e77080cc845a7e6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2e472179723f4aa0803da70c4299466f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/03370b7bee704702b05777c9cf566bfc] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=83.9 K 2023-04-22 19:55:55,653 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/5e019f4a0a6f44cbb3c9952cda27e701, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1ce6a0aa20f443819689b2dd92ff625c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/e257c1d05d6f4295bba16138e04e0051, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/6aa4f8ef15d74e999eb061ea86038a20, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bd6b5abfac434e00bf08be959bf6f6d2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bbbccc995cb541999e60f9b2a66f2116, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/a8452519ef0d4efd8f60127a335417de] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=81.5 K 2023-04-22 19:55:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,653 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,653 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 9660a007f5644680a1be7a919b852261, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682193348470 2023-04-22 19:55:55,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,654 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 5e019f4a0a6f44cbb3c9952cda27e701, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682193348470 2023-04-22 19:55:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,654 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,654 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 93edf50fbce241c0b99c82b4c660ab12, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682193348514 2023-04-22 19:55:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,654 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1ce6a0aa20f443819689b2dd92ff625c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682193348514 2023-04-22 19:55:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,654 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 9236676d203c44f2a95bd2162e667fa9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1682193350641 2023-04-22 19:55:55,654 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting e257c1d05d6f4295bba16138e04e0051, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1682193350641 2023-04-22 19:55:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,655 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 8846415461d44d1ba45ded2619946ea2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682193351003 2023-04-22 19:55:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,655 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 6aa4f8ef15d74e999eb061ea86038a20, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682193351003 2023-04-22 19:55:55,655 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 2d5f56e2d6f34a549e77080cc845a7e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682193352139 2023-04-22 19:55:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,655 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting bd6b5abfac434e00bf08be959bf6f6d2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682193352139 2023-04-22 19:55:55,655 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,655 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 2e472179723f4aa0803da70c4299466f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682193353360 2023-04-22 19:55:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,655 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting bbbccc995cb541999e60f9b2a66f2116, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682193353360 2023-04-22 19:55:55,656 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 03370b7bee704702b05777c9cf566bfc, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682193354459 2023-04-22 19:55:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,656 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a8452519ef0d4efd8f60127a335417de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682193354459 2023-04-22 19:55:55,657 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,657 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,657 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,657 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,657 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:55,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:55,657 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,658 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,658 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,658 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,658 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,658 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:55,658 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:55,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,667 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#B#compaction#405 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:55,668 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#A#compaction#404 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:55,684 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/e29cb332d64d47d48da9aeff65e58524 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/e29cb332d64d47d48da9aeff65e58524 2023-04-22 19:55:55,684 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/80acd121bac3484da29d3b7c1487c794 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/80acd121bac3484da29d3b7c1487c794 2023-04-22 19:55:55,688 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/A of 62116471b8dabdbd8f57094f112778d1 into e29cb332d64d47d48da9aeff65e58524(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:55,688 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:55,688 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/A, priority=9, startTime=1682193355651; duration=0sec 2023-04-22 19:55:55,688 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:55,688 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 19:55:55,689 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/B of 62116471b8dabdbd8f57094f112778d1 into 80acd121bac3484da29d3b7c1487c794(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:55,689 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:55,689 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/B, priority=9, startTime=1682193355651; duration=0sec 2023-04-22 19:55:55,689 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:55,690 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83502 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 19:55:55,690 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/C is initiating minor compaction (all files) 2023-04-22 19:55:55,690 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/C in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,690 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ef7d9ab3d9e3405fa4fa94d244ac41db, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/812604ed7d514be6a5a75259b298de1b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/c8887bb66e714b3b9911c3771bdb2caf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/1f0a46d20a7444cab37eb691df7dbd93, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d5bc1cfb6b4a4379a73411180f1c723c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ebb268245b824ab6b3943a744ecdefa5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/67deee18b02e49769f9d9b2868fc62b5] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=81.5 K 2023-04-22 19:55:55,690 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting ef7d9ab3d9e3405fa4fa94d244ac41db, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682193348470 2023-04-22 19:55:55,691 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 812604ed7d514be6a5a75259b298de1b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682193348514 2023-04-22 19:55:55,691 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c8887bb66e714b3b9911c3771bdb2caf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1682193350641 2023-04-22 19:55:55,691 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 1f0a46d20a7444cab37eb691df7dbd93, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682193351003 2023-04-22 19:55:55,692 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting d5bc1cfb6b4a4379a73411180f1c723c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682193352139 2023-04-22 19:55:55,692 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting ebb268245b824ab6b3943a744ecdefa5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682193353360 2023-04-22 19:55:55,692 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 67deee18b02e49769f9d9b2868fc62b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682193354459 2023-04-22 19:55:55,702 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#C#compaction#406 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:55,758 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:55,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:55,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:55,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:55,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:55,762 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:55,762 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,763 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:55,763 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:55,763 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,763 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,767 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,767 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,768 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,768 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:55,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:55,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:55,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:55,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:55,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:55,769 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:55,769 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,769 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:55,769 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 19:55:55,770 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:55,770 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:55,770 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:55,770 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:55,770 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:55,770 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:55,782 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/ab3fd162c27648908b1b37a896087ada 2023-04-22 19:55:55,793 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/7be696c57e46452a87447b42041d0a67 2023-04-22 19:55:55,805 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/68f26ce9bff8473d8e46f13c41af89a8 2023-04-22 19:55:55,809 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/ab3fd162c27648908b1b37a896087ada as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/ab3fd162c27648908b1b37a896087ada 2023-04-22 19:55:55,813 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/ab3fd162c27648908b1b37a896087ada, entries=150, sequenceid=170, filesize=11.8 K 2023-04-22 19:55:55,814 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/7be696c57e46452a87447b42041d0a67 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/7be696c57e46452a87447b42041d0a67 2023-04-22 19:55:55,817 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/7be696c57e46452a87447b42041d0a67, entries=150, sequenceid=170, filesize=11.8 K 2023-04-22 19:55:55,818 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/68f26ce9bff8473d8e46f13c41af89a8 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/68f26ce9bff8473d8e46f13c41af89a8 2023-04-22 19:55:55,821 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/68f26ce9bff8473d8e46f13c41af89a8, entries=150, sequenceid=170, filesize=11.8 K 2023-04-22 19:55:55,822 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 62116471b8dabdbd8f57094f112778d1 in 53ms, sequenceid=170, compaction requested=false 2023-04-22 19:55:55,822 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:55,822 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:55,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:55,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:55,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:55,824 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,824 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,825 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,825 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:55,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:55,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b6159c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:55,826 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:55,826 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,827 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,827 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:55,828 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,828 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:55,828 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:55,828 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:55,829 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,830 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:55,830 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,830 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:55,832 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:55,832 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:55,832 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:55,832 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:55,832 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:55,835 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,835 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,835 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,836 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,838 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,842 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:55,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:55,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:55,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,842 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:55,842 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:55,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,943 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:55,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:55,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:55,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:55,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:55,946 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:55,946 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,946 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:55,946 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:55,946 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,946 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,947 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,947 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,948 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,948 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:55,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:55,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:55,949 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,949 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:55,950 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:55,950 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:55,951 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,951 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,952 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,952 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:55,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dad058f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:55,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:55,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,955 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:55,955 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,955 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:55,956 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:55,956 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,956 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:55,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:55,959 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:55,959 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:55,959 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:55,959 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:55,959 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:55,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:55,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,961 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:55,962 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:55,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:55,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:55,966 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:55,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:55,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:55,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:55,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:55,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:55,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:55,966 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:55,966 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:55,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:55,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:56,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:56,019 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:55:56,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:56,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:56,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:56,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:56,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:56,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:56,040 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/0083291a33a94f5da99174c4e454c9ad 2023-04-22 19:55:56,052 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/3bd339a3f307437bbb71e956abbad25c 2023-04-22 19:55:56,067 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:56,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:56,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:56,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:56,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:56,070 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:56,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:56,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:56,071 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,071 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/5c2ea1267721436bb457a2b756ddc44e 2023-04-22 19:55:56,073 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:56,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:56,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:56,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:56,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:56,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/0083291a33a94f5da99174c4e454c9ad as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0083291a33a94f5da99174c4e454c9ad 2023-04-22 19:55:56,078 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,078 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,078 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,078 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:56,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@758e4756[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:56,080 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:56,080 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0083291a33a94f5da99174c4e454c9ad, entries=150, sequenceid=181, filesize=11.8 K 2023-04-22 19:55:56,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:56,081 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,081 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:56,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/3bd339a3f307437bbb71e956abbad25c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/3bd339a3f307437bbb71e956abbad25c 2023-04-22 19:55:56,083 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,083 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,083 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,083 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:56,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:56,086 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:56,086 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:56,086 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:56,086 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:56,086 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:56,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/3bd339a3f307437bbb71e956abbad25c, entries=100, sequenceid=181, filesize=9.4 K 2023-04-22 19:55:56,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/5c2ea1267721436bb457a2b756ddc44e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5c2ea1267721436bb457a2b756ddc44e 2023-04-22 19:55:56,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,092 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5c2ea1267721436bb457a2b756ddc44e, entries=100, sequenceid=181, filesize=9.4 K 2023-04-22 19:55:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,095 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=93.93 KB/96180 for 62116471b8dabdbd8f57094f112778d1 in 76ms, sequenceid=181, compaction requested=true 2023-04-22 19:55:56,095 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:56,095 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:56,095 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-22 19:55:56,095 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-22 19:55:56,095 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 7 compacting, 2 eligible, 16 blocking 2023-04-22 19:55:56,096 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 19:55:56,096 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 19:55:56,096 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. because compaction request was cancelled 2023-04-22 19:55:56,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,096 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:55:56,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,096 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:56,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:56,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,096 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,097 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,097 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,097 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,097 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:56,097 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:56,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:56,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,097 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36344 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:55:56,097 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/A is initiating minor compaction (all files) 2023-04-22 19:55:56,098 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/A in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,098 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/e29cb332d64d47d48da9aeff65e58524, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/ab3fd162c27648908b1b37a896087ada, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0083291a33a94f5da99174c4e454c9ad] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=35.5 K 2023-04-22 19:55:56,098 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting e29cb332d64d47d48da9aeff65e58524, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682193354459 2023-04-22 19:55:56,098 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ab3fd162c27648908b1b37a896087ada, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1682193354834 2023-04-22 19:55:56,098 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0083291a33a94f5da99174c4e454c9ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682193356010 2023-04-22 19:55:56,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:56,103 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 19:55:56,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:56,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:56,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:56,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:56,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:56,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:56,103 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#A#compaction#413 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:56,121 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/7cc0b5a8d134400f9f829fa8854d6b25 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/7cc0b5a8d134400f9f829fa8854d6b25 2023-04-22 19:55:56,126 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/C of 62116471b8dabdbd8f57094f112778d1 into 7cc0b5a8d134400f9f829fa8854d6b25(size=12.0 K), total size for store is 33.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:56,126 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:56,126 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/C, priority=9, startTime=1682193355651; duration=0sec 2023-04-22 19:55:56,127 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:56,127 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:55:56,128 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:55:56,128 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/B is initiating minor compaction (all files) 2023-04-22 19:55:56,128 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/B in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,129 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/80acd121bac3484da29d3b7c1487c794, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/7be696c57e46452a87447b42041d0a67, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/3bd339a3f307437bbb71e956abbad25c] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=33.2 K 2023-04-22 19:55:56,129 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 80acd121bac3484da29d3b7c1487c794, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682193354459 2023-04-22 19:55:56,129 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 7be696c57e46452a87447b42041d0a67, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1682193354834 2023-04-22 19:55:56,130 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 3bd339a3f307437bbb71e956abbad25c, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682193356010 2023-04-22 19:55:56,131 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/69740598d7a243678ef3983c9a16cf86 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/69740598d7a243678ef3983c9a16cf86 2023-04-22 19:55:56,138 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#B#compaction#415 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:56,142 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/A of 62116471b8dabdbd8f57094f112778d1 into 69740598d7a243678ef3983c9a16cf86(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:56,143 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:56,143 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/A, priority=13, startTime=1682193356095; duration=0sec 2023-04-22 19:55:56,143 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:56,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:56,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193416141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:56,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193416144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,158 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/597714bd4c3b43c193ff33199a550a61 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/597714bd4c3b43c193ff33199a550a61 2023-04-22 19:55:56,165 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/B of 62116471b8dabdbd8f57094f112778d1 into 597714bd4c3b43c193ff33199a550a61(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:56,165 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:56,165 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/B, priority=13, startTime=1682193356095; duration=0sec 2023-04-22 19:55:56,166 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:56,197 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:56,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:56,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:56,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:56,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:56,200 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:56,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:56,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:56,201 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,201 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,202 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,202 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,202 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:56,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:56,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:56,204 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,204 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:56,204 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:56,204 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:56,209 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@370f01bd[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:56,214 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:56,214 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,215 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,215 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:56,215 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:56,215 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,215 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:56,215 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:56,218 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:56,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:56,221 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:56,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:56,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:56,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:56,221 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:56,224 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,224 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,224 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,224 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,227 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,229 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,230 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:56,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:56,230 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,230 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,230 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:56,230 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:56,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:56,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193416252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:56,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193416254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,330 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:56,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:56,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:56,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:56,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:56,334 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:56,334 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,334 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:56,334 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:56,334 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,335 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,336 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,336 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,336 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,336 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:56,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:56,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:56,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:56,337 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,337 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:56,337 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:56,338 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,338 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:56,338 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:56,338 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,338 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:56,338 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:56,339 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,339 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:56,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a7f835[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:56,341 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:56,341 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,348 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:56,348 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,348 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:56,348 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:56,348 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:56,350 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:56,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:56,352 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:56,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:56,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:56,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:56,352 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:56,354 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,354 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,354 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,356 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,358 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,359 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:56,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:56,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,360 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:56,360 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:56,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:56,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193416454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,460 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:56,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:56,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:56,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:56,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:56,463 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:56,463 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,463 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:56,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:56,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,466 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,466 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:56,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193416463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,466 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,467 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:56,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:56,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:56,468 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,468 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:56,468 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:56,468 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,469 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,470 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,470 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@230473[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:56,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:56,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:56,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:56,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:56,473 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,473 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:56,474 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,474 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:56,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:56,477 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:56,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:56,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:56,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:56,477 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:56,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,481 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,482 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:56,485 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:56,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:56,485 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:56,485 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:56,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/e3777abf89444a369bd93b3ece4bbb5b 2023-04-22 19:55:56,585 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:56,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:56,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:56,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:56,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:56,589 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:56,589 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,589 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:56,589 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:56,589 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,590 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,591 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,591 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,591 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,591 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:56,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:56,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:56,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:56,593 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,593 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:56,593 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:56,593 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:56,595 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79047d6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:56,598 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:56,599 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,601 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:56,605 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:56,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,605 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,605 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:56,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:56,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:56,605 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:56,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:56,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:56,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:56,605 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:56,608 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,608 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,609 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,609 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,611 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,614 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:56,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:56,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,615 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:56,615 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:56,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:56,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40190 deadline: 1682193416710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,715 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:56,715 DEBUG [Thread-2961] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:55:56,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:56,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:56,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:56,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:56,718 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:56,719 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,719 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:56,719 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:56,719 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,719 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,721 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,721 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,722 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,722 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:56,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:56,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:56,723 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,723 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:56,723 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:56,723 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:56,725 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,725 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,725 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,725 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@485dac70[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:56,726 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:56,726 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,729 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:56,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:56,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:56,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:56,730 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:56,730 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,730 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:56,730 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:56,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:56,733 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:56,733 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:56,733 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:56,733 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:56,733 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:56,735 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,735 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,735 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,736 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,737 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,740 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:56,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:56,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:56,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,740 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:56,740 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:56,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,740 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:56,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193416761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:56,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193416769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,840 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:56,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:56,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:56,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:56,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:56,844 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:56,844 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,844 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:56,844 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:56,844 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,844 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,845 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,845 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,846 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,846 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:56,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:56,847 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,847 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:56,847 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:56,847 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,848 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:56,848 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:56,848 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,848 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:56,848 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,849 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:56,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7201831e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:56,851 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:56,851 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,853 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:56,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:56,853 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:56,853 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,853 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:56,854 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:56,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:56,857 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:56,857 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:56,857 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:56,857 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:56,857 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:56,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,859 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,860 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,862 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,862 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,862 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,862 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:56,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,863 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:56,863 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:56,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/8f68e83319624aeb9071d7acdedc35b4 2023-04-22 19:55:56,963 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:56,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:56,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:56,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:56,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:56,967 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:56,967 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,967 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:56,967 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:56,967 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,967 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,971 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,971 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,972 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,972 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:56,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:56,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:56,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:56,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:56,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:56,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:56,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:56,973 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,973 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:56,973 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:56,973 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:56,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:56,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:56,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:56,973 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:56,974 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,974 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,975 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,975 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:56,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a1901b1[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:56,978 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:56,978 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,978 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:56,979 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,979 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:56,980 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:56,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:56,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:56,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:56,982 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:56,982 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:56,982 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:56,982 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:56,982 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:56,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,987 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:56,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:56,989 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,991 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:56,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:56,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:56,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:56,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:56,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:56,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:56,993 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:56,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:56,993 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:56,993 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:56,993 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:56,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,094 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:57,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:57,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:57,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:57,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:57,097 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:57,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:57,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:57,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,099 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,099 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,099 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,099 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:57,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:57,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:57,101 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:57,101 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,101 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:57,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:57,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:57,102 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,102 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:57,102 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:57,102 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:57,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:57,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:57,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:57,104 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,104 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,105 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:57,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f3245d3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:57,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:57,106 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,107 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,107 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:57,107 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:57,107 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:57,107 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,107 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:57,108 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,108 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,108 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,108 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:57,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:57,110 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:57,110 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:57,110 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:57,110 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:57,110 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:57,112 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,112 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,112 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,112 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,113 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,116 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,116 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,116 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,116 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,116 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,116 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:57,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:57,116 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,116 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,116 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,117 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:57,117 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:57,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,217 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:57,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:57,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:57,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:57,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:57,220 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:57,220 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,220 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:57,220 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:57,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,222 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,222 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,222 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,222 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:57,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:57,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:57,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:57,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:57,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:57,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:57,224 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,224 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:57,224 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:57,224 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:57,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:57,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:57,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:57,225 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,225 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,226 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:57,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a295a69[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:57,227 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:57,227 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,228 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:57,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:57,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:57,229 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,229 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:57,231 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:57,231 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,231 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,231 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:57,234 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:57,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:57,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:57,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:57,234 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:57,235 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,235 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,236 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,236 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,237 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:57,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,239 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:57,240 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,239 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,240 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:57,240 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:57,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,240 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:57,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:57,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193417268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:57,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193417280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,340 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:57,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:57,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:57,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:57,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:57,343 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:57,343 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,343 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:57,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:57,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,345 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,345 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,345 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,346 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:57,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:57,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:57,347 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,347 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:57,347 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:57,347 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:57,348 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:57,348 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,348 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:57,348 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:57,350 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:57,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34ce3941[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:57,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:57,352 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,353 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:57,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:57,353 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:57,353 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:57,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:57,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,355 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:57,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:57,358 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:57,358 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:57,358 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:57,358 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:57,358 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:57,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,360 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,361 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,362 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,367 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,367 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,367 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,367 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,367 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,368 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:57,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:57,368 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,368 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:57,368 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:57,368 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:57,368 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,368 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/5a99ba7afb984df58f30f8511553cb98 2023-04-22 19:55:57,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/e3777abf89444a369bd93b3ece4bbb5b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/e3777abf89444a369bd93b3ece4bbb5b 2023-04-22 19:55:57,379 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/e3777abf89444a369bd93b3ece4bbb5b, entries=150, sequenceid=200, filesize=11.8 K 2023-04-22 19:55:57,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/8f68e83319624aeb9071d7acdedc35b4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/8f68e83319624aeb9071d7acdedc35b4 2023-04-22 19:55:57,382 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/8f68e83319624aeb9071d7acdedc35b4, entries=150, sequenceid=200, filesize=11.8 K 2023-04-22 19:55:57,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/5a99ba7afb984df58f30f8511553cb98 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5a99ba7afb984df58f30f8511553cb98 2023-04-22 19:55:57,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5a99ba7afb984df58f30f8511553cb98, entries=150, sequenceid=200, filesize=11.8 K 2023-04-22 19:55:57,387 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 62116471b8dabdbd8f57094f112778d1 in 1284ms, sequenceid=200, compaction requested=true 2023-04-22 19:55:57,387 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:57,387 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:57,387 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:57,387 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-22 19:55:57,388 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-22 19:55:57,387 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:57,388 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 19:55:57,388 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 19:55:57,388 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 19:55:57,388 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 19:55:57,388 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. because compaction request was cancelled 2023-04-22 19:55:57,388 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. because compaction request was cancelled 2023-04-22 19:55:57,389 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:55:57,390 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45990 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:55:57,390 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/C is initiating minor compaction (all files) 2023-04-22 19:55:57,390 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/C in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,390 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/7cc0b5a8d134400f9f829fa8854d6b25, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/68f26ce9bff8473d8e46f13c41af89a8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5c2ea1267721436bb457a2b756ddc44e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5a99ba7afb984df58f30f8511553cb98] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=44.9 K 2023-04-22 19:55:57,391 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 7cc0b5a8d134400f9f829fa8854d6b25, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682193354459 2023-04-22 19:55:57,392 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 68f26ce9bff8473d8e46f13c41af89a8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1682193354834 2023-04-22 19:55:57,392 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 5c2ea1267721436bb457a2b756ddc44e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682193356010 2023-04-22 19:55:57,392 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 5a99ba7afb984df58f30f8511553cb98, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682193356086 2023-04-22 19:55:57,402 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#C#compaction#418 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:57,468 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:57,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:57,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:57,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:57,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:57,471 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:57,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:57,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:57,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,473 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:57,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:57,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:57,476 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:57,476 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 19:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:57,483 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/454e370bdea640a7b741cf9bb8ad6cbf 2023-04-22 19:55:57,494 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/dd6e1bf4c64547d1ba659fce76796c69 2023-04-22 19:55:57,505 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/d415b7eee5bb4b0a8f3358bd9e5b4e44 2023-04-22 19:55:57,509 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/454e370bdea640a7b741cf9bb8ad6cbf as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/454e370bdea640a7b741cf9bb8ad6cbf 2023-04-22 19:55:57,512 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/454e370bdea640a7b741cf9bb8ad6cbf, entries=150, sequenceid=216, filesize=11.8 K 2023-04-22 19:55:57,513 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/dd6e1bf4c64547d1ba659fce76796c69 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/dd6e1bf4c64547d1ba659fce76796c69 2023-04-22 19:55:57,516 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/dd6e1bf4c64547d1ba659fce76796c69, entries=150, sequenceid=216, filesize=11.8 K 2023-04-22 19:55:57,516 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/d415b7eee5bb4b0a8f3358bd9e5b4e44 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d415b7eee5bb4b0a8f3358bd9e5b4e44 2023-04-22 19:55:57,521 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d415b7eee5bb4b0a8f3358bd9e5b4e44, entries=150, sequenceid=216, filesize=11.8 K 2023-04-22 19:55:57,522 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 62116471b8dabdbd8f57094f112778d1 in 46ms, sequenceid=216, compaction requested=true 2023-04-22 19:55:57,522 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:57,522 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:57,524 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:57,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:57,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f8b2e0[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:57,526 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:57,526 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,527 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:57,528 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,529 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:57,529 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,531 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:57,531 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,531 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:57,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:57,532 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:57,532 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:57,532 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:57,532 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:57,532 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:57,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,534 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,534 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,537 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,539 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,539 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,539 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,539 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:57,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:57,539 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,539 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:57,540 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:57,540 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:57,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,539 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,540 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,540 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,540 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:57,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:55:57,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:57,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:57,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:57,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:57,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:57,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:57,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193357587, max timestamp=1682193357631Num uniques 859; 2023-04-22 19:55:57,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:55:57,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193357587, max timestamp=1682193357631Num uniques 859; 2023-04-22 19:55:57,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193357587, max timestamp=1682193357631Num uniques 859; 2023-04-22 19:55:57,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:55:57,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:55:57,640 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:57,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:57,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:57,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:57,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:57,643 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:57,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:57,643 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:57,644 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,644 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,646 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,646 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,647 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,647 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:57,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:57,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:57,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:57,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:57,648 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:57,649 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,649 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:57,649 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:57,649 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:57,651 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:57,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b196131[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:57,654 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:57,654 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,656 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,656 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:57,657 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:57,658 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:57,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:57,660 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:57,660 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:57,660 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:57,660 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:57,660 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:57,664 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,664 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,664 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,664 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,665 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,667 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,667 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,667 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,667 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,668 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:57,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:57,667 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:57,668 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:57,668 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:57,668 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,668 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,668 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,668 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,668 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:57,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193417711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:57,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193417712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,768 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:57,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:57,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:57,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:57,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:57,772 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:57,772 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,772 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:57,772 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:57,773 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,773 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,774 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,774 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,774 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,774 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:57,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:57,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:57,775 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:57,775 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:57,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:57,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:57,776 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,776 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:57,776 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:57,776 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:57,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:57,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:57,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:57,777 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,778 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:57,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6be38429[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:57,779 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:57,779 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,781 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,782 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:57,782 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:57,782 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:57,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,782 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:57,783 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,783 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:57,783 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,783 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:57,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:57,787 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:57,787 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:57,787 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:57,787 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:57,787 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:57,788 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,788 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,788 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,789 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,790 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:57,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:57,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,792 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,793 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,792 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,793 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:57,793 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:57,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:57,818 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/0f07b9f7e546492399039ba636704c8e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/0f07b9f7e546492399039ba636704c8e 2023-04-22 19:55:57,823 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/C of 62116471b8dabdbd8f57094f112778d1 into 0f07b9f7e546492399039ba636704c8e(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:57,823 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:57,823 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/C, priority=12, startTime=1682193357387; duration=0sec 2023-04-22 19:55:57,823 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:57,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:57,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193417829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:57,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193417829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,893 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:57,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:57,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:57,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:57,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:57,898 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:57,898 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,898 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:57,899 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:57,899 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,899 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,903 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,903 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,904 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,904 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:57,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:57,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:57,906 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,906 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:57,906 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:57,906 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:57,907 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,907 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,908 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,908 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:57,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e22bc25[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:57,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:57,910 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:57,911 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,911 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:57,914 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:57,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:57,918 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:57,918 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:57,918 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:57,918 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:57,918 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:57,919 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:57,919 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,919 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:57,920 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:57,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,921 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:57,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:57,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:57,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:57,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:57,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:57,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:57,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:57,923 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:57,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:57,923 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:57,924 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:57,924 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:57,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:57,977 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/a6939a98f1914b1e8983d5561539b61a 2023-04-22 19:55:58,024 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:58,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:58,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:58,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:58,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:58,029 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:58,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:58,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:58,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,030 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,030 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,031 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,031 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:58,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:58,032 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,032 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:58,032 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:58,033 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d1e74b3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:58,044 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:58,045 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:58,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193418035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:58,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193418036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,046 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:58,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:58,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:58,046 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,046 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:58,047 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:58,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:58,051 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:58,051 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:58,051 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:58,051 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:58,051 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:58,052 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,052 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,052 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,053 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,055 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,058 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:58,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,058 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:58,058 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:58,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:58,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,158 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:58,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:58,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:58,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:58,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:58,162 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:58,162 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,162 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:58,162 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:58,162 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,162 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,163 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,163 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,163 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,164 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:58,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:58,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:58,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:58,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:58,165 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:58,167 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dae92c8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:58,169 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:58,169 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,170 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:58,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:58,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:58,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:58,175 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:58,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:58,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:58,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:58,175 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:58,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,177 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,178 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,180 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,180 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:58,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:58,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,180 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,181 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:58,181 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:58,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,281 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:58,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:58,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:58,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:58,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:58,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:58,286 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:58,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193418283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:58,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:58,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,286 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,287 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,287 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,288 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:58,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:58,289 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,289 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:58,289 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:58,289 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:58,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:58,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193418289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,294 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:58,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a1b9798[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:58,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:58,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,297 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,298 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:58,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:58,298 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:58,298 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:58,299 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:58,299 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,299 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,299 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:58,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:58,304 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:58,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:58,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:58,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:58,304 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:58,307 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,307 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,307 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,309 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,313 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,313 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,313 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,313 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,313 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,313 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,314 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,314 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,314 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,314 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:58,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:58,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,314 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:58,314 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:58,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:58,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:58,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193418346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:58,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193418347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/9686781a1db841bdaf2366dc5f670888 2023-04-22 19:55:58,414 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:58,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:58,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:58,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:58,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:58,417 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:58,417 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,417 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:58,417 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:58,418 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,418 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,419 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,419 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:58,421 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:58,421 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:58,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:58,421 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:58,421 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,421 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:58,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:58,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:58,422 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,422 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:58,422 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:58,422 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:58,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:58,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:58,422 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:58,423 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:58,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1163c56a[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:58,426 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:58,426 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,428 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:58,428 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:58,428 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:58,428 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,428 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,429 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:58,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:58,433 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:58,433 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:58,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:58,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:58,434 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:58,435 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,435 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,435 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,436 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,438 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,442 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,442 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,442 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:58,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:58,442 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,442 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,442 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,442 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,442 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,442 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,442 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,442 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:58,443 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:58,543 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:58,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:58,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:58,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:58,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:58,546 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:58,546 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,546 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:58,546 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:58,547 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,547 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,549 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,549 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,549 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,549 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:58,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:58,553 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,553 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:58,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd68019[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:58,556 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:58,556 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,557 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:58,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:58,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:58,558 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,558 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:58,559 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:58,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:58,562 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:58,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:58,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:58,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:58,562 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:58,565 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,566 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,566 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,572 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,574 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:58,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:58,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,574 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:58,574 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:58,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,674 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:58,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:58,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:58,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:58,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:58,678 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:58,678 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,678 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:58,678 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:58,678 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,679 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,680 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,680 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:58,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:58,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:58,682 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,682 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:58,682 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:58,682 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:58,684 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:58,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36c779f9[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:58,686 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:58,686 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,688 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,688 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:58,688 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:58,688 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:58,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:58,691 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:58,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:58,694 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:58,694 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:58,694 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:58,694 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:58,694 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:58,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,696 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,697 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,700 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:58,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:58,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,700 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:58,700 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:58,800 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:58,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:58,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/8508adc92a9142c781bab23e559e0cb3 2023-04-22 19:55:58,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:58,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:58,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:58,804 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:58,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:58,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:58,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,806 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,806 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,807 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/a6939a98f1914b1e8983d5561539b61a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/a6939a98f1914b1e8983d5561539b61a 2023-04-22 19:55:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,807 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,807 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:58,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:58,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:58,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:58,809 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:58,809 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,809 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:58,809 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:58,809 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:58,809 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,810 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:58,810 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:58,810 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:58,811 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/a6939a98f1914b1e8983d5561539b61a, entries=200, sequenceid=226, filesize=14.1 K 2023-04-22 19:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:58,812 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/9686781a1db841bdaf2366dc5f670888 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9686781a1db841bdaf2366dc5f670888 2023-04-22 19:55:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:58,813 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9686781a1db841bdaf2366dc5f670888, entries=150, sequenceid=226, filesize=11.8 K 2023-04-22 19:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40969fee[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:58,816 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:58,816 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,816 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/8508adc92a9142c781bab23e559e0cb3 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/8508adc92a9142c781bab23e559e0cb3 2023-04-22 19:55:58,817 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:58,817 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,817 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:58,820 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,820 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/8508adc92a9142c781bab23e559e0cb3, entries=150, sequenceid=226, filesize=11.8 K 2023-04-22 19:55:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,821 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 62116471b8dabdbd8f57094f112778d1 in 1255ms, sequenceid=226, compaction requested=true 2023-04-22 19:55:58,821 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,821 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:58,821 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:55:58,821 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,821 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:55:58,821 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,822 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50876 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:55:58,822 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:55:58,822 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/A is initiating minor compaction (all files) 2023-04-22 19:55:58,823 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/B is initiating minor compaction (all files) 2023-04-22 19:55:58,823 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/A in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,823 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/B in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,823 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/69740598d7a243678ef3983c9a16cf86, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/e3777abf89444a369bd93b3ece4bbb5b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/454e370bdea640a7b741cf9bb8ad6cbf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/a6939a98f1914b1e8983d5561539b61a] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=49.7 K 2023-04-22 19:55:58,823 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/597714bd4c3b43c193ff33199a550a61, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/8f68e83319624aeb9071d7acdedc35b4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/dd6e1bf4c64547d1ba659fce76796c69, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9686781a1db841bdaf2366dc5f670888] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=47.3 K 2023-04-22 19:55:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:58,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:58,823 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:58,823 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:58,823 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:58,823 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:58,823 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:58,823 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 69740598d7a243678ef3983c9a16cf86, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682193354834 2023-04-22 19:55:58,823 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 597714bd4c3b43c193ff33199a550a61, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682193354834 2023-04-22 19:55:58,824 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting e3777abf89444a369bd93b3ece4bbb5b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682193356086 2023-04-22 19:55:58,824 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 8f68e83319624aeb9071d7acdedc35b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682193356086 2023-04-22 19:55:58,824 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 454e370bdea640a7b741cf9bb8ad6cbf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682193356136 2023-04-22 19:55:58,824 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting dd6e1bf4c64547d1ba659fce76796c69, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682193356136 2023-04-22 19:55:58,825 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting a6939a98f1914b1e8983d5561539b61a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682193357552 2023-04-22 19:55:58,825 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 9686781a1db841bdaf2366dc5f670888, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682193357552 2023-04-22 19:55:58,826 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,827 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,829 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,833 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:58,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:58,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:58,834 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:58,834 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:58,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,836 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#B#compaction#426 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:58,836 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#A#compaction#425 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:58,855 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/fc709c3b6fe44362bff2012dab0618de as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/fc709c3b6fe44362bff2012dab0618de 2023-04-22 19:55:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:55:58,857 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-22 19:55:58,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:58,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:58,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:58,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:58,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:58,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:55:58,859 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/907617583a4848849953227eb740d1a2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/907617583a4848849953227eb740d1a2 2023-04-22 19:55:58,860 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/A of 62116471b8dabdbd8f57094f112778d1 into fc709c3b6fe44362bff2012dab0618de(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:58,860 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:58,860 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/A, priority=12, startTime=1682193358821; duration=0sec 2023-04-22 19:55:58,860 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:55:58,860 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:55:58,861 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:55:58,861 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/C is initiating minor compaction (all files) 2023-04-22 19:55:58,861 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/C in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,862 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/0f07b9f7e546492399039ba636704c8e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d415b7eee5bb4b0a8f3358bd9e5b4e44, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/8508adc92a9142c781bab23e559e0cb3] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=35.6 K 2023-04-22 19:55:58,862 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 0f07b9f7e546492399039ba636704c8e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682193356086 2023-04-22 19:55:58,862 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting d415b7eee5bb4b0a8f3358bd9e5b4e44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682193356136 2023-04-22 19:55:58,863 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 8508adc92a9142c781bab23e559e0cb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682193357552 2023-04-22 19:55:58,864 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/B of 62116471b8dabdbd8f57094f112778d1 into 907617583a4848849953227eb740d1a2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:58,864 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:58,864 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/B, priority=12, startTime=1682193358821; duration=0sec 2023-04-22 19:55:58,864 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:58,871 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#C#compaction#428 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:55:58,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/b32ecd9ac2694168a18da42b67b5e882 2023-04-22 19:55:58,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:58,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193418882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:58,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193418883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,934 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:58,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:58,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:58,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:58,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:58,937 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:58,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:58,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:58,938 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,938 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,939 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:58,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:58,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:58,941 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:58,941 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:58,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:58,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:58,942 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,942 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:58,942 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:58,942 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:58,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:58,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:58,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:58,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:58,943 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,943 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:58,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f514a03[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:58,945 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:58,945 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,947 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:58,948 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,948 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:58,949 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:58,951 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:58,951 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:58,951 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:58,951 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:58,951 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:58,953 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:58,953 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,953 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:58,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,954 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,956 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:58,956 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:58,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:58,956 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:58,956 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:58,956 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,957 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:58,957 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,957 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:58,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:58,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:58,957 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:58,957 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:58,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:58,957 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:58,957 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:58,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:58,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193418987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:58,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:58,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193418991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,057 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:59,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:59,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:59,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:59,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:59,061 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:59,061 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,061 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:59,061 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:59,061 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,061 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,063 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,063 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:59,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:59,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:59,065 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:59,065 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,065 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:59,065 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:59,065 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:59,065 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,065 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:59,066 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:59,066 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:59,069 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,069 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,069 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,069 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64accdb8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:59,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:59,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,071 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,072 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:59,072 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:59,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:59,072 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:59,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:59,073 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:59,075 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:59,075 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:59,076 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:59,076 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:59,076 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:59,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,082 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,083 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,085 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,085 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,085 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,085 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,085 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,085 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:59,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:59,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:59,085 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,085 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:59,085 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:59,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,085 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,085 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,086 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,185 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:59,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:59,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:59,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:59,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:59,188 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:59,189 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,189 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:59,189 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:59,189 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,189 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,190 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,190 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,190 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,190 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:59,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:59,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:59,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:59,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:59,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:59,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:59,192 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,192 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:59,192 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:59,192 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:59,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:59,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:59,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:59,193 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,193 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,194 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,194 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:59,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193419192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:59,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69296865[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:59,196 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:59,196 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,198 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,198 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:59,198 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:59,198 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,198 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:59,198 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:59,200 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,200 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,200 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,200 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:59,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193419197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:59,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:59,203 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:59,203 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:59,203 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:59,203 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:59,203 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:59,204 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,205 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,205 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,205 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,206 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,208 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:59,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:59,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,209 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,209 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:59,209 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:59,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,288 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/5c2577c870d541279e0bb6e80846f352 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5c2577c870d541279e0bb6e80846f352 2023-04-22 19:55:59,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/bea51f271e214027bab2eb949b21ef60 2023-04-22 19:55:59,294 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/C of 62116471b8dabdbd8f57094f112778d1 into 5c2577c870d541279e0bb6e80846f352(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:55:59,294 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:59,294 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/C, priority=13, startTime=1682193358821; duration=0sec 2023-04-22 19:55:59,294 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:55:59,309 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:59,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:59,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:59,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:59,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:59,312 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:59,312 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,312 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:59,312 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:59,313 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,313 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,315 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,315 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,315 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,315 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:59,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:59,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:59,316 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,316 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:59,317 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:59,317 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:59,318 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,318 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,318 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,318 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1db4e341[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:59,321 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:59,321 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,321 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:59,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:59,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:59,323 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,323 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:59,323 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:59,324 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:59,324 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:59,324 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:59,324 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:59,324 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:59,326 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,326 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,326 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,326 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,327 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:59,330 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:59,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:59,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,330 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:59,330 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:59,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,430 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:59,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:59,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:59,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:59,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:59,434 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:59,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:59,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:59,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,435 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,436 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,436 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,436 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:59,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:59,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:59,437 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,438 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:59,438 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:59,438 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:59,439 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,439 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:59,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f0a3a6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:59,441 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:59,441 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,443 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:59,445 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,445 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:59,445 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,445 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:59,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:59,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:59,449 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:59,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:59,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:59,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:59,449 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:59,451 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,451 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,451 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,451 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,454 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,460 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,460 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,460 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,460 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,461 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:59,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:59,461 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,461 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,461 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,461 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,461 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:59,461 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:59,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193419496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:55:59,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1682193419503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,561 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:59,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:59,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:59,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:59,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:59,565 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:59,565 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,565 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:59,565 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:59,565 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,565 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,566 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,567 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,567 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:59,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:59,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:59,568 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:59,568 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,568 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:59,568 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:59,568 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:59,568 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,568 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:59,568 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:59,568 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,568 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:59,568 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:59,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:59,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:59,570 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:59,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:59,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab3740d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:59,572 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:59,572 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,573 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,573 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,573 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:59,573 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:59,573 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:59,573 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:59,576 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:59,577 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,577 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:59,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:59,579 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:59,579 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:59,579 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:59,579 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:59,579 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:59,581 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,581 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,581 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,582 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,583 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,585 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,585 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:59,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:59,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,587 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:59,587 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:59,687 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:59,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:59,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:59,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:59,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:59,690 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:59,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:59,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:59,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,692 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,692 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,692 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,692 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:59,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:59,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:59,695 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,695 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:59,695 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:55:59,695 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:55:59,697 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,697 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,698 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,698 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fb557e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:55:59,699 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:55:59,700 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,701 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,701 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:55:59,701 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:55:59,701 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:55:59,701 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/bf8e2a1fc63a4dd889b8352192fa87bc 2023-04-22 19:55:59,701 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,701 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:55:59,702 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:55:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:55:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/b32ecd9ac2694168a18da42b67b5e882 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/b32ecd9ac2694168a18da42b67b5e882 2023-04-22 19:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:55:59,706 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:55:59,706 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:55:59,707 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:55:59,707 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:55:59,707 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:55:59,708 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,708 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,708 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:55:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:55:59,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/b32ecd9ac2694168a18da42b67b5e882, entries=200, sequenceid=264, filesize=14.3 K 2023-04-22 19:55:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,709 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:55:59,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/bea51f271e214027bab2eb949b21ef60 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bea51f271e214027bab2eb949b21ef60 2023-04-22 19:55:59,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:55:59,711 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:55:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:55:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,716 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,716 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:55:59,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,716 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:55:59,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:55:59,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:55:59,716 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:55:59,716 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:55:59,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:55:59,717 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,718 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,718 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,718 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,718 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,718 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:55:59,718 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:55:59,718 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bea51f271e214027bab2eb949b21ef60, entries=150, sequenceid=264, filesize=11.9 K 2023-04-22 19:55:59,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/bf8e2a1fc63a4dd889b8352192fa87bc as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/bf8e2a1fc63a4dd889b8352192fa87bc 2023-04-22 19:55:59,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/bf8e2a1fc63a4dd889b8352192fa87bc, entries=150, sequenceid=264, filesize=11.9 K 2023-04-22 19:55:59,732 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 62116471b8dabdbd8f57094f112778d1 in 875ms, sequenceid=264, compaction requested=false 2023-04-22 19:55:59,732 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:55:59,817 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:55:59,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:55:59,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:55:59,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:55:59,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:55:59,820 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:55:59,820 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,820 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:55:59,820 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:55:59,821 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,821 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,825 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:55:59,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:55:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:55:59,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:55:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:55:59,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:55:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:55:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:55:59,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:55:59,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:55:59,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:55:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:55:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:55:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:55:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:55:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:55:59,832 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:55:59,832 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:55:59,833 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 19:55:59,833 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:55:59,833 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:59,833 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:55:59,833 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:55:59,833 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:55:59,833 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:00,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:00,008 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:00,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193360030, max timestamp=1682193360059Num uniques 859; 2023-04-22 19:56:00,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193360030, max timestamp=1682193360059Num uniques 859; 2023-04-22 19:56:00,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193360030, max timestamp=1682193360059Num uniques 859; 2023-04-22 19:56:00,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:56:00,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:56:00,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:56:00,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:00,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193420115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:00,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:00,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193420116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:00,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:00,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40156 deadline: 1682193420224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:00,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:00,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193420224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:00,241 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/0b50bc19929b4b2aaa250c7456d36b54 2023-04-22 19:56:00,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:00,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193420297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:00,302 DEBUG [Thread-2963] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:56:00,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:00,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193420299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:00,306 DEBUG [Thread-2965] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:56:00,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:00,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193420427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:00,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:00,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193420428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:00,653 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/db7c5e2092794131abc24ceb9314a721 2023-04-22 19:56:00,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:00,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193420732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:00,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:00,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193420736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,063 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/796fd1c9b2424f2b8cf87b66571efcfe 2023-04-22 19:56:01,067 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/0b50bc19929b4b2aaa250c7456d36b54 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0b50bc19929b4b2aaa250c7456d36b54 2023-04-22 19:56:01,070 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0b50bc19929b4b2aaa250c7456d36b54, entries=150, sequenceid=276, filesize=11.9 K 2023-04-22 19:56:01,071 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/db7c5e2092794131abc24ceb9314a721 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/db7c5e2092794131abc24ceb9314a721 2023-04-22 19:56:01,074 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/db7c5e2092794131abc24ceb9314a721, entries=150, sequenceid=276, filesize=11.9 K 2023-04-22 19:56:01,075 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/796fd1c9b2424f2b8cf87b66571efcfe as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/796fd1c9b2424f2b8cf87b66571efcfe 2023-04-22 19:56:01,080 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/796fd1c9b2424f2b8cf87b66571efcfe, entries=150, sequenceid=276, filesize=11.9 K 2023-04-22 19:56:01,080 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 62116471b8dabdbd8f57094f112778d1 in 1247ms, sequenceid=276, compaction requested=true 2023-04-22 19:56:01,080 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:01,081 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:01,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:01,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:01,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:01,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:01,082 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,082 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:01,083 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,083 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:01,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:01,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@426f1fcb[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:01,084 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:01,084 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,085 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:01,086 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:01,086 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:01,086 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,086 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:01,088 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:01,088 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:01,088 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:01,088 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:01,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:01,091 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:01,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:01,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:01,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:01,091 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:01,093 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,093 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,093 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:01,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:01,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:01,093 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:01,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:01,095 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:01,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:01,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:01,102 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:01,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:01,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:01,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:01,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:01,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:01,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:01,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,102 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:01,102 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:01,102 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:01,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:01,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:01,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:01,202 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:01,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:01,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:01,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:01,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:01,206 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:01,206 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:01,206 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:01,206 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:01,206 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,206 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:01,208 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:01,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:01,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:01,208 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,208 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:01,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:01,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:01,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:01,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:01,210 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:01,210 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:01,210 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:01,210 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:01,210 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:01,210 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:01,210 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:01,210 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-22 19:56:01,211 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:01,211 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:01,211 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:01,211 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:01,211 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:01,211 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:01,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:01,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:01,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:01,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193421262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:01,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193421263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:01,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193421367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193421370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:01,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193421575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:01,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193421577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,619 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/60f9a2db08d5454fbe1adb9129a61160 2023-04-22 19:56:01,629 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/26d4c85f8b6a432084bcc847877c114e 2023-04-22 19:56:01,641 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/af92df713b6b41daa0cb345c33effa2a 2023-04-22 19:56:01,645 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/60f9a2db08d5454fbe1adb9129a61160 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/60f9a2db08d5454fbe1adb9129a61160 2023-04-22 19:56:01,649 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/60f9a2db08d5454fbe1adb9129a61160, entries=150, sequenceid=313, filesize=11.9 K 2023-04-22 19:56:01,650 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/26d4c85f8b6a432084bcc847877c114e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/26d4c85f8b6a432084bcc847877c114e 2023-04-22 19:56:01,654 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/26d4c85f8b6a432084bcc847877c114e, entries=150, sequenceid=313, filesize=11.9 K 2023-04-22 19:56:01,654 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/af92df713b6b41daa0cb345c33effa2a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/af92df713b6b41daa0cb345c33effa2a 2023-04-22 19:56:01,672 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/af92df713b6b41daa0cb345c33effa2a, entries=150, sequenceid=313, filesize=11.9 K 2023-04-22 19:56:01,673 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 62116471b8dabdbd8f57094f112778d1 in 463ms, sequenceid=313, compaction requested=true 2023-04-22 19:56:01,673 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:01,673 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:01,686 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,686 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:01,686 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:01,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:01,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fdf69b3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:01,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:01,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,689 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,689 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:01,689 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:01,689 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:01,691 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:01,691 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:01,691 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:01,691 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,691 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:01,691 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:01,691 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:01,691 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:01,691 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:01,691 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:01,691 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:01,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:01,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:01,695 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:01,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:01,697 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:01,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:01,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:01,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:01,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:01,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:01,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:01,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:01,699 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:01,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:01,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:01,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:01,700 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:01,700 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:01,800 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:01,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:01,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:01,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:01,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:01,804 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:01,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:01,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:01,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:01,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:01,805 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:01,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:01,806 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:01,806 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:01,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:01,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:01,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:01,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:01,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:01,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:01,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:01,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:01,807 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:01,808 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:01,808 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 19:56:01,808 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:01,808 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:01,808 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:01,808 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:01,808 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:01,808 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:01,817 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/d5ed5df59c424df1935b76ef2e17cafb 2023-04-22 19:56:01,830 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/fde7f7107df5447c89902a4ce45f81fd 2023-04-22 19:56:01,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:01,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:01,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193361919, max timestamp=1682193361957Num uniques 859; 2023-04-22 19:56:01,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:01,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193361919, max timestamp=1682193361957Num uniques 859; 2023-04-22 19:56:01,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193361919, max timestamp=1682193361957Num uniques 859; 2023-04-22 19:56:01,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:01,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:02,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:02,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193422023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193361995, max timestamp=1682193362026Num uniques 1018; 2023-04-22 19:56:02,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 19:56:02,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193361995, max timestamp=1682193362026Num uniques 1018; 2023-04-22 19:56:02,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 19:56:02,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193361995, max timestamp=1682193362026Num uniques 1018; 2023-04-22 19:56:02,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 19:56:02,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682193362096, max timestamp=1682193362129Num uniques 1027; 2023-04-22 19:56:02,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 19:56:02,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682193362096, max timestamp=1682193362129Num uniques 1027; 2023-04-22 19:56:02,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682193362096, max timestamp=1682193362129Num uniques 1027; 2023-04-22 19:56:02,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 19:56:02,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 19:56:02,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682193362165, max timestamp=1682193362198Num uniques 1036; 2023-04-22 19:56:02,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-22 19:56:02,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682193362165, max timestamp=1682193362198Num uniques 1036; 2023-04-22 19:56:02,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-22 19:56:02,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682193362165, max timestamp=1682193362198Num uniques 1036; 2023-04-22 19:56:02,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-22 19:56:02,240 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/d7bfa6e5327d4ee19d34bc8009238e36 2023-04-22 19:56:02,244 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/d5ed5df59c424df1935b76ef2e17cafb as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d5ed5df59c424df1935b76ef2e17cafb 2023-04-22 19:56:02,248 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d5ed5df59c424df1935b76ef2e17cafb, entries=100, sequenceid=322, filesize=9.5 K 2023-04-22 19:56:02,248 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/fde7f7107df5447c89902a4ce45f81fd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fde7f7107df5447c89902a4ce45f81fd 2023-04-22 19:56:02,252 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fde7f7107df5447c89902a4ce45f81fd, entries=100, sequenceid=322, filesize=9.5 K 2023-04-22 19:56:02,253 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/d7bfa6e5327d4ee19d34bc8009238e36 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d7bfa6e5327d4ee19d34bc8009238e36 2023-04-22 19:56:02,256 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d7bfa6e5327d4ee19d34bc8009238e36, entries=100, sequenceid=322, filesize=9.5 K 2023-04-22 19:56:02,257 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=114.05 KB/116790 for 62116471b8dabdbd8f57094f112778d1 in 449ms, sequenceid=322, compaction requested=true 2023-04-22 19:56:02,257 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:02,257 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:02,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:02,258 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=120.76 KB heapSize=333.23 KB 2023-04-22 19:56:02,259 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:02,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:02,259 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,259 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,259 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:02,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:02,259 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,259 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:02,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:02,259 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0d6559[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:02,261 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:02,261 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,264 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,264 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:02,264 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:02,264 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:02,264 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,264 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:02,265 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:02,266 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:02,266 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:02,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/5e95fbf5b6f04660a85060bb4dffcf10 2023-04-22 19:56:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:02,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:02,276 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:02,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:02,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:02,276 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:02,276 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:02,278 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,278 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:02,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,278 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:02,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,279 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,280 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,290 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,290 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:02,290 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:02,290 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,290 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,290 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,290 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,290 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:02,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:02,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:02,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:02,291 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:02,291 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:02,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:02,291 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:02,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:02,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:02,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/807eb5214f344c5f8d928c1cf54b113f 2023-04-22 19:56:02,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/b55547c342e449f99652a018ff2d159f 2023-04-22 19:56:02,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/5e95fbf5b6f04660a85060bb4dffcf10 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/5e95fbf5b6f04660a85060bb4dffcf10 2023-04-22 19:56:02,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:02,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193422314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/5e95fbf5b6f04660a85060bb4dffcf10, entries=200, sequenceid=406, filesize=14.3 K 2023-04-22 19:56:02,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/807eb5214f344c5f8d928c1cf54b113f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/807eb5214f344c5f8d928c1cf54b113f 2023-04-22 19:56:02,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/807eb5214f344c5f8d928c1cf54b113f, entries=150, sequenceid=406, filesize=11.9 K 2023-04-22 19:56:02,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:02,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/b55547c342e449f99652a018ff2d159f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/b55547c342e449f99652a018ff2d159f 2023-04-22 19:56:02,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193422315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/b55547c342e449f99652a018ff2d159f, entries=150, sequenceid=406, filesize=11.9 K 2023-04-22 19:56:02,341 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~333.19 KB/341184, currentSize=60.38 KB/61830 for 62116471b8dabdbd8f57094f112778d1 in 83ms, sequenceid=406, compaction requested=true 2023-04-22 19:56:02,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:02,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:02,342 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 19:56:02,342 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 19:56:02,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:02,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:02,344 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71000 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 19:56:02,344 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 75880 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 19:56:02,344 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/B is initiating minor compaction (all files) 2023-04-22 19:56:02,344 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/A is initiating minor compaction (all files) 2023-04-22 19:56:02,344 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/B in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:02,344 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/A in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:02,344 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/907617583a4848849953227eb740d1a2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bea51f271e214027bab2eb949b21ef60, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/db7c5e2092794131abc24ceb9314a721, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/26d4c85f8b6a432084bcc847877c114e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fde7f7107df5447c89902a4ce45f81fd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/807eb5214f344c5f8d928c1cf54b113f] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=69.3 K 2023-04-22 19:56:02,344 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/fc709c3b6fe44362bff2012dab0618de, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/b32ecd9ac2694168a18da42b67b5e882, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0b50bc19929b4b2aaa250c7456d36b54, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/60f9a2db08d5454fbe1adb9129a61160, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d5ed5df59c424df1935b76ef2e17cafb, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/5e95fbf5b6f04660a85060bb4dffcf10] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=74.1 K 2023-04-22 19:56:02,344 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 907617583a4848849953227eb740d1a2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682193357552 2023-04-22 19:56:02,345 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting fc709c3b6fe44362bff2012dab0618de, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682193357552 2023-04-22 19:56:02,345 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting bea51f271e214027bab2eb949b21ef60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682193357685 2023-04-22 19:56:02,345 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting b32ecd9ac2694168a18da42b67b5e882, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682193357685 2023-04-22 19:56:02,346 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting db7c5e2092794131abc24ceb9314a721, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1682193358870 2023-04-22 19:56:02,346 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 0b50bc19929b4b2aaa250c7456d36b54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1682193358870 2023-04-22 19:56:02,346 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 26d4c85f8b6a432084bcc847877c114e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682193360105 2023-04-22 19:56:02,347 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting fde7f7107df5447c89902a4ce45f81fd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682193361258 2023-04-22 19:56:02,347 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 807eb5214f344c5f8d928c1cf54b113f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682193362235 2023-04-22 19:56:02,348 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 60f9a2db08d5454fbe1adb9129a61160, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682193360105 2023-04-22 19:56:02,351 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting d5ed5df59c424df1935b76ef2e17cafb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682193361258 2023-04-22 19:56:02,351 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 5e95fbf5b6f04660a85060bb4dffcf10, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682193362217 2023-04-22 19:56:02,377 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#A#compaction#443 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:02,378 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#B#compaction#444 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:02,391 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:02,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:02,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:02,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:02,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:02,402 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:02,402 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:02,403 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:02,403 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:02,403 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,403 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:02,404 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/f2e639070ba140ed9006da2b2add255a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/f2e639070ba140ed9006da2b2add255a 2023-04-22 19:56:02,405 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:02,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:02,405 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:02,407 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,407 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:02,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:02,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:02,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:02,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:02,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:02,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:02,408 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:02,409 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:02,409 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:02,409 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:02,409 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:02,409 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 19:56:02,410 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:02,410 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:02,410 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:02,410 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:02,410 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:02,410 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:02,410 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/B of 62116471b8dabdbd8f57094f112778d1 into f2e639070ba140ed9006da2b2add255a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:02,411 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:02,411 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/B, priority=10, startTime=1682193362342; duration=0sec 2023-04-22 19:56:02,411 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:02,411 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 19:56:02,412 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71000 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 19:56:02,413 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/C is initiating minor compaction (all files) 2023-04-22 19:56:02,413 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/C in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:02,413 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5c2577c870d541279e0bb6e80846f352, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/bf8e2a1fc63a4dd889b8352192fa87bc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/796fd1c9b2424f2b8cf87b66571efcfe, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/af92df713b6b41daa0cb345c33effa2a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d7bfa6e5327d4ee19d34bc8009238e36, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/b55547c342e449f99652a018ff2d159f] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=69.3 K 2023-04-22 19:56:02,415 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 5c2577c870d541279e0bb6e80846f352, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682193357552 2023-04-22 19:56:02,415 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting bf8e2a1fc63a4dd889b8352192fa87bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682193357685 2023-04-22 19:56:02,416 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 796fd1c9b2424f2b8cf87b66571efcfe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1682193358870 2023-04-22 19:56:02,416 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting af92df713b6b41daa0cb345c33effa2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682193360105 2023-04-22 19:56:02,416 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting d7bfa6e5327d4ee19d34bc8009238e36, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682193361258 2023-04-22 19:56:02,416 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b55547c342e449f99652a018ff2d159f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682193362235 2023-04-22 19:56:02,428 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/d72ed41aed5542228bfac85c6f0cbcb2 2023-04-22 19:56:02,429 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#C#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:02,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:02,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:02,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:02,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193422486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:02,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193422493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:02,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193422594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:02,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193422606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,807 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/ab0b0631a4af47f78eafea675a52712b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/ab0b0631a4af47f78eafea675a52712b 2023-04-22 19:56:02,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:02,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193422803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,811 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/A of 62116471b8dabdbd8f57094f112778d1 into ab0b0631a4af47f78eafea675a52712b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:02,811 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:02,811 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/A, priority=10, startTime=1682193362342; duration=0sec 2023-04-22 19:56:02,811 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:02,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:02,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193422813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,840 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/7f08091a2899460b9e058318ef4fc3f2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/7f08091a2899460b9e058318ef4fc3f2 2023-04-22 19:56:02,844 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/C of 62116471b8dabdbd8f57094f112778d1 into 7f08091a2899460b9e058318ef4fc3f2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:02,844 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:02,844 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/C, priority=10, startTime=1682193362342; duration=0sec 2023-04-22 19:56:02,844 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:02,846 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/9a97aad1893e43d2a3da0d8b4dcd56f5 2023-04-22 19:56:02,857 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/198764ff047048bea529b7849293da20 2023-04-22 19:56:02,863 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/d72ed41aed5542228bfac85c6f0cbcb2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d72ed41aed5542228bfac85c6f0cbcb2 2023-04-22 19:56:02,866 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d72ed41aed5542228bfac85c6f0cbcb2, entries=150, sequenceid=418, filesize=11.9 K 2023-04-22 19:56:02,866 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/9a97aad1893e43d2a3da0d8b4dcd56f5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9a97aad1893e43d2a3da0d8b4dcd56f5 2023-04-22 19:56:02,870 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9a97aad1893e43d2a3da0d8b4dcd56f5, entries=150, sequenceid=418, filesize=11.9 K 2023-04-22 19:56:02,871 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/198764ff047048bea529b7849293da20 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/198764ff047048bea529b7849293da20 2023-04-22 19:56:02,874 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/198764ff047048bea529b7849293da20, entries=150, sequenceid=418, filesize=11.9 K 2023-04-22 19:56:02,875 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 62116471b8dabdbd8f57094f112778d1 in 466ms, sequenceid=418, compaction requested=false 2023-04-22 19:56:02,875 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:02,875 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:02,875 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:02,875 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:02,875 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,875 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:02,875 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:02,876 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,876 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:02,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,877 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:02,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:02,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cff8b43[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:02,880 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:02,880 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,881 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,881 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:02,881 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:02,881 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:02,881 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,882 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:02,883 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:02,883 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,883 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:02,883 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:02,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:02,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:02,886 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:02,886 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:02,886 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:02,886 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:02,886 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:02,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,889 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,890 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:02,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:02,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,891 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:02,892 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,894 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,894 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:02,895 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:02,895 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:02,895 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:02,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:02,895 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:02,895 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:02,895 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,895 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:02,895 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:02,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:02,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:02,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:02,895 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:02,895 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:02,995 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:02,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:02,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:03,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:03,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:03,000 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:03,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:03,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:03,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:03,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:03,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:03,001 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:03,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:03,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:03,001 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:03,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,002 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:03,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:03,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:03,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:03,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:03,003 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:03,003 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:03,003 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:03,003 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:03,003 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:03,003 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:03,003 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:03,004 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 19:56:03,004 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:03,004 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:03,004 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:03,004 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:03,004 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:03,004 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:03,014 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/78d4c47c07b24272b6d4be4f08d69d5b 2023-04-22 19:56:03,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:03,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:03,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:03,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193423139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:03,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193423144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:03,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193423247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:03,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193423247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,431 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/fbf87316beb74cc2908cf7db6fa57948 2023-04-22 19:56:03,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:03,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193423450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:03,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193423451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:03,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193423759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:03,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193423759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,842 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/f6a9887f2b7042848ef3cdd47d946fe9 2023-04-22 19:56:03,846 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/78d4c47c07b24272b6d4be4f08d69d5b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/78d4c47c07b24272b6d4be4f08d69d5b 2023-04-22 19:56:03,849 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/78d4c47c07b24272b6d4be4f08d69d5b, entries=150, sequenceid=441, filesize=11.9 K 2023-04-22 19:56:03,850 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/fbf87316beb74cc2908cf7db6fa57948 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fbf87316beb74cc2908cf7db6fa57948 2023-04-22 19:56:03,854 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fbf87316beb74cc2908cf7db6fa57948, entries=150, sequenceid=441, filesize=11.9 K 2023-04-22 19:56:03,855 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/f6a9887f2b7042848ef3cdd47d946fe9 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6a9887f2b7042848ef3cdd47d946fe9 2023-04-22 19:56:03,858 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6a9887f2b7042848ef3cdd47d946fe9, entries=150, sequenceid=441, filesize=11.9 K 2023-04-22 19:56:03,859 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 62116471b8dabdbd8f57094f112778d1 in 856ms, sequenceid=441, compaction requested=true 2023-04-22 19:56:03,859 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:03,859 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:03,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:03,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:03,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:03,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:03,864 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,864 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:03,864 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:03,864 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:03,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:03,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46545250[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:03,866 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:03,866 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:03,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:03,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:03,870 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:03,870 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:03,870 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:03,870 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,870 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:03,873 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:03,873 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,873 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:03,873 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:03,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:03,875 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:03,875 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:03,876 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:03,876 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:03,876 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:03,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:03,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:03,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:03,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:03,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:03,878 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:03,880 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:03,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:03,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:03,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:03,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:03,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:03,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:03,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:03,882 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:03,882 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:03,882 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:03,882 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,883 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:03,883 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:03,883 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,883 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:03,883 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:03,883 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:03,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:03,982 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:03,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:03,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:03,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:03,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:03,986 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:03,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:03,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:03,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:03,987 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:03,987 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:03,990 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,990 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:03,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:03,990 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:03,990 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:03,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:03,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:03,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:03,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:03,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:03,991 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:03,993 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:03,993 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:03,993 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 19:56:03,993 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:03,993 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:03,994 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:03,994 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:03,994 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:03,994 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:04,001 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/c3c02928892e4f7a90194d0da54c2c63 2023-04-22 19:56:04,021 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/85d4811681604799aaa6944f7d217b0a 2023-04-22 19:56:04,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:04,272 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:04,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193424321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193424325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40140 deadline: 1682193424326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193424326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,336 DEBUG [Thread-2965] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:56:04,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193424427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193424435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193424435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,441 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/0b718003ce1b461e90da1dbab1fde5ea 2023-04-22 19:56:04,445 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/c3c02928892e4f7a90194d0da54c2c63 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/c3c02928892e4f7a90194d0da54c2c63 2023-04-22 19:56:04,449 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/c3c02928892e4f7a90194d0da54c2c63, entries=150, sequenceid=453, filesize=11.9 K 2023-04-22 19:56:04,449 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/85d4811681604799aaa6944f7d217b0a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/85d4811681604799aaa6944f7d217b0a 2023-04-22 19:56:04,453 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/85d4811681604799aaa6944f7d217b0a, entries=150, sequenceid=453, filesize=11.9 K 2023-04-22 19:56:04,454 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/0b718003ce1b461e90da1dbab1fde5ea as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/0b718003ce1b461e90da1dbab1fde5ea 2023-04-22 19:56:04,457 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/0b718003ce1b461e90da1dbab1fde5ea, entries=150, sequenceid=453, filesize=11.9 K 2023-04-22 19:56:04,458 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 62116471b8dabdbd8f57094f112778d1 in 465ms, sequenceid=453, compaction requested=true 2023-04-22 19:56:04,458 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:04,458 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:04,458 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:04,458 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:04,458 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,458 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:04,458 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:04,459 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:04,459 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:04,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:04,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@222239d8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:04,461 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:04,461 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:04,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:04,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:04,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:04,462 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:04,462 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:04,462 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:04,463 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,463 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:04,464 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,464 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:04,464 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:04,464 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:04,466 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:04,466 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:04,466 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:04,466 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:04,466 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:04,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:04,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:04,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:04,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:04,469 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:04,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:04,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:04,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:04,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:04,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:04,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:04,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:04,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:04,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:04,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:04,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:04,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:04,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,472 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,472 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:04,472 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:04,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:04,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:04,472 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:04,472 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:04,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:04,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:04,472 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:04,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:04,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:04,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:04,473 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:04,473 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:04,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:04,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:04,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:04,573 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:04,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:04,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:04,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:04,577 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:04,577 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:04,577 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:04,577 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:04,577 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:04,577 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:04,578 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:04,578 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:04,578 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,578 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:04,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:04,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:04,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:04,580 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:04,580 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:04,580 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:04,580 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:04,580 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:04,580 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:04,580 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:04,580 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 19:56:04,580 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:04,580 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:04,580 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:04,580 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:04,581 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:04,581 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:04,588 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/83ddd8eba0e84e1bbfa1c0417c582196 2023-04-22 19:56:04,598 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/4ac5d536eca94aeeb65402ceec9ef473 2023-04-22 19:56:04,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:04,641 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:04,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40116 deadline: 1682193424656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193424657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193424664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193424766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193424767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193424768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193424972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193424973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:04,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:04,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193424974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,008 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/f6db5840cb8948a79c50bc8a9fa7574e 2023-04-22 19:56:05,011 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/83ddd8eba0e84e1bbfa1c0417c582196 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/83ddd8eba0e84e1bbfa1c0417c582196 2023-04-22 19:56:05,014 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/83ddd8eba0e84e1bbfa1c0417c582196, entries=150, sequenceid=473, filesize=11.9 K 2023-04-22 19:56:05,015 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/4ac5d536eca94aeeb65402ceec9ef473 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/4ac5d536eca94aeeb65402ceec9ef473 2023-04-22 19:56:05,018 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/4ac5d536eca94aeeb65402ceec9ef473, entries=150, sequenceid=473, filesize=11.9 K 2023-04-22 19:56:05,019 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/f6db5840cb8948a79c50bc8a9fa7574e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6db5840cb8948a79c50bc8a9fa7574e 2023-04-22 19:56:05,022 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6db5840cb8948a79c50bc8a9fa7574e, entries=150, sequenceid=473, filesize=11.9 K 2023-04-22 19:56:05,022 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 62116471b8dabdbd8f57094f112778d1 in 442ms, sequenceid=473, compaction requested=true 2023-04-22 19:56:05,023 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:05,023 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:05,024 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,024 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:05,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,025 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:05,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:05,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17b0a768[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:05,026 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:05,026 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,029 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:05,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,029 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:05,030 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:05,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:05,030 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:05,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:05,032 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:05,032 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:05,032 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:05,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:05,033 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:05,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:05,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:05,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:05,034 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:05,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:05,035 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:05,036 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,038 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:05,038 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:05,038 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:05,038 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:05,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:05,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:05,038 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:05,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:05,038 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:05,038 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:05,038 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,038 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:05,038 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,039 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:05,038 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:05,139 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:05,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:05,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:05,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:05,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:05,142 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:05,142 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:05,142 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:05,142 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:05,143 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:05,143 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:05,144 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,145 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:05,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:05,145 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,145 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:05,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:05,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:05,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:05,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:05,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:05,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:05,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:05,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:05,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:05,147 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:05,147 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:05,147 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 19:56:05,147 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:05,147 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:05,147 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:05,148 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:05,148 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:05,148 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:05,155 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/fc6ea7ce06044420a03e2733b1b25d5b 2023-04-22 19:56:05,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:05,281 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:05,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193425321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193425322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193425323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193425430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193425436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193425437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,566 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/c7c113a7de424bc681089b1cb4e4c4c0 2023-04-22 19:56:05,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193425637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193425643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193425644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193425944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193425950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:05,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193425952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,978 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/13e5ae975d2d4c4e847595e52ab2a733 2023-04-22 19:56:05,982 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/fc6ea7ce06044420a03e2733b1b25d5b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/fc6ea7ce06044420a03e2733b1b25d5b 2023-04-22 19:56:05,986 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/fc6ea7ce06044420a03e2733b1b25d5b, entries=150, sequenceid=485, filesize=11.9 K 2023-04-22 19:56:05,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/c7c113a7de424bc681089b1cb4e4c4c0 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/c7c113a7de424bc681089b1cb4e4c4c0 2023-04-22 19:56:05,990 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/c7c113a7de424bc681089b1cb4e4c4c0, entries=150, sequenceid=485, filesize=11.9 K 2023-04-22 19:56:05,990 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/13e5ae975d2d4c4e847595e52ab2a733 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/13e5ae975d2d4c4e847595e52ab2a733 2023-04-22 19:56:05,994 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/13e5ae975d2d4c4e847595e52ab2a733, entries=150, sequenceid=485, filesize=11.9 K 2023-04-22 19:56:05,994 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 62116471b8dabdbd8f57094f112778d1 in 847ms, sequenceid=485, compaction requested=true 2023-04-22 19:56:05,994 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:05,994 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:05,994 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:05,994 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:05,995 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,995 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:05,995 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:05,996 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,996 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:05,997 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,997 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:05,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:05,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1795934e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:05,998 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:05,998 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,999 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:05,999 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:05,999 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:05,999 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:05,999 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,000 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:06,002 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:06,002 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:06,002 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:06,002 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:06,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:06,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:06,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:06,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:06,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:06,004 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:06,004 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:06,004 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:06,004 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:06,005 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:06,006 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:06,006 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:06,006 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:06,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:06,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:06,007 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,008 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:06,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:06,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:06,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:06,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:06,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:06,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:06,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:06,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:06,010 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:06,011 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:06,011 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:06,010 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:06,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:06,111 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:06,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:06,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:06,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:06,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:06,114 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:06,115 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:06,115 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:06,115 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:06,115 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:06,115 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:06,116 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:06,116 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:06,117 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,117 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:06,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:06,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:06,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:06,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:06,118 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:06,118 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:06,118 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:06,118 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:06,118 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:06,118 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:06,118 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:06,118 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 19:56:06,119 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:06,119 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:06,119 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:06,119 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:06,119 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:06,119 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:06,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:06,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:06,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:06,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193426479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:06,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:06,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193426487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193426488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,529 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/cf27704d56344b6f94a3ee16857c2e38 2023-04-22 19:56:06,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:06,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193426589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:06,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193426597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:06,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193426597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:06,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40190 deadline: 1682193426754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,758 DEBUG [Thread-2961] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18224 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:56:06,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:06,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193426797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:06,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193426806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:06,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193426807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:06,938 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/1457d564d7af44e58ab1b8db70c4ea70 2023-04-22 19:56:07,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:07,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193427105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:07,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193427113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:07,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193427114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,348 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/de2fada3043e48f7bbc049cbd3dcdd93 2023-04-22 19:56:07,353 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/cf27704d56344b6f94a3ee16857c2e38 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/cf27704d56344b6f94a3ee16857c2e38 2023-04-22 19:56:07,358 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/cf27704d56344b6f94a3ee16857c2e38, entries=150, sequenceid=505, filesize=11.9 K 2023-04-22 19:56:07,359 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/1457d564d7af44e58ab1b8db70c4ea70 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1457d564d7af44e58ab1b8db70c4ea70 2023-04-22 19:56:07,362 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1457d564d7af44e58ab1b8db70c4ea70, entries=150, sequenceid=505, filesize=11.9 K 2023-04-22 19:56:07,363 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/de2fada3043e48f7bbc049cbd3dcdd93 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/de2fada3043e48f7bbc049cbd3dcdd93 2023-04-22 19:56:07,366 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/de2fada3043e48f7bbc049cbd3dcdd93, entries=150, sequenceid=505, filesize=11.9 K 2023-04-22 19:56:07,367 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 62116471b8dabdbd8f57094f112778d1 in 1249ms, sequenceid=505, compaction requested=true 2023-04-22 19:56:07,367 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:07,367 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:07,367 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:07,367 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:07,367 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,368 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:07,368 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:07,375 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,375 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:07,375 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,375 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:07,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:07,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73b7953e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:07,377 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:07,377 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,378 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,378 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:07,378 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:07,378 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:07,378 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,378 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:07,379 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,379 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:07,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:07,382 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:07,382 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:07,382 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:07,382 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:07,382 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:07,384 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,384 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:07,384 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:07,385 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,386 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:07,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,390 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:07,390 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:07,390 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:07,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:07,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:07,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:07,390 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:07,390 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:07,390 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:07,390 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,390 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:07,390 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,390 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:07,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:07,390 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,390 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,490 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:07,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:07,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:07,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:07,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:07,494 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:07,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:07,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:07,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:07,496 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,496 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:07,497 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:07,497 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:07,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:07,497 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,497 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:07,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:07,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:07,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:07,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:07,503 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:07,503 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:07,503 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:07,503 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:07,504 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 19:56:07,504 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:07,504 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:07,504 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:07,504 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:07,504 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:07,504 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:07,513 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/47c46c0059e44e71a0d5b5771ddee4da 2023-04-22 19:56:07,525 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/b563ed304dfd4b5fb3d450b56ed0d809 2023-04-22 19:56:07,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:07,619 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:07,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:07,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193427667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:07,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193427672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:07,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193427673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:07,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193427774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:07,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193427787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:07,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193427787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,936 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/a4548e37f37a4fee8f341ab8ea7b4e74 2023-04-22 19:56:07,940 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/47c46c0059e44e71a0d5b5771ddee4da as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/47c46c0059e44e71a0d5b5771ddee4da 2023-04-22 19:56:07,944 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/47c46c0059e44e71a0d5b5771ddee4da, entries=150, sequenceid=517, filesize=11.9 K 2023-04-22 19:56:07,944 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/b563ed304dfd4b5fb3d450b56ed0d809 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/b563ed304dfd4b5fb3d450b56ed0d809 2023-04-22 19:56:07,948 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/b563ed304dfd4b5fb3d450b56ed0d809, entries=150, sequenceid=517, filesize=11.9 K 2023-04-22 19:56:07,948 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/a4548e37f37a4fee8f341ab8ea7b4e74 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/a4548e37f37a4fee8f341ab8ea7b4e74 2023-04-22 19:56:07,954 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/a4548e37f37a4fee8f341ab8ea7b4e74, entries=150, sequenceid=517, filesize=11.9 K 2023-04-22 19:56:07,955 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 62116471b8dabdbd8f57094f112778d1 in 452ms, sequenceid=517, compaction requested=true 2023-04-22 19:56:07,955 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:07,955 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:07,955 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:07,955 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:07,955 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,955 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:07,955 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:07,957 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:07,957 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:07,957 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,957 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:07,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:07,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40e88e69[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:07,958 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:07,958 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,959 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,960 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:07,960 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:07,960 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:07,960 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,960 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:07,962 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:07,962 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,962 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,962 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:07,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:07,964 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:07,964 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:07,964 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:07,964 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:07,964 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:07,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:07,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:07,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:07,966 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:07,967 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,969 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,969 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:07,969 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:07,969 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:07,969 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:07,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:07,969 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:07,969 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:07,969 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:07,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:07,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:07,969 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:07,969 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:07,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:07,969 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:07,970 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:07,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:07,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:07,996 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-22 19:56:07,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:07,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:07,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:07,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:07,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:07,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:08,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/155f313a703d43bba94306451d6c5ddd 2023-04-22 19:56:08,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:08,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193428022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:08,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193428023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:08,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193428026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,070 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:08,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:08,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:08,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:08,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:08,073 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:08,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:08,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:08,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:08,074 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,074 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,075 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:08,075 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,075 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,076 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:08,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:08,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:08,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:08,077 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:08,077 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:08,077 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:08,077 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:08,079 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:08,080 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:08,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,081 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:08,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:08,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52f1fafb[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:08,083 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:08,083 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,084 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,084 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:08,084 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:08,084 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:08,084 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,084 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:08,085 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,085 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:08,085 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:08,085 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:08,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:08,088 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:08,088 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:08,088 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:08,088 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:08,088 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:08,093 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,093 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:08,093 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:08,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:08,094 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:08,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,095 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,097 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,097 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,097 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:08,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:08,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:08,097 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:08,097 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:08,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:08,098 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:08,098 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,098 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:08,098 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:08,098 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,098 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,098 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:08,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:08,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:08,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193428137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:08,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193428137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:08,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193428144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,198 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:08,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:08,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:08,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:08,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:08,202 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:08,202 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:08,202 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:08,202 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:08,202 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,202 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,203 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,203 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:08,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:08,204 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,204 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:08,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:08,205 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:08,205 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:08,205 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:08,205 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:08,205 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:08,205 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:08,205 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:08,206 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:08,206 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:08,207 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,207 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:08,208 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,208 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c51c08[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:08,209 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:08,209 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,211 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:08,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:08,211 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:08,211 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,211 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:08,214 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,214 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:08,214 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:08,214 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:08,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:08,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:08,217 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:08,217 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:08,217 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:08,217 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:08,217 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:08,218 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,218 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:08,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,218 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:08,219 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,220 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:08,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:08,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:08,223 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:08,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:08,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:08,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,223 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,223 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:08,223 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:08,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:08,324 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:08,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:08,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:08,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:08,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:08,327 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:08,327 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:08,327 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:08,327 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:08,328 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,328 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,329 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:08,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:08,329 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,329 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:08,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:08,331 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:08,331 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. started... 2023-04-22 19:56:08,331 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. as already flushing 2023-04-22 19:56:08,331 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,333 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:08,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26146a6b[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:08,337 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:08,337 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,338 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,338 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:08,338 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:08,338 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:08,338 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,338 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:08,340 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,340 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:08,341 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:08,341 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:08,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:08,343 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:08,343 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:08,343 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:08,343 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:08,343 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:08,344 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:08,344 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,345 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:08,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:08,345 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,346 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:08,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:08,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:08,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:08,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:08,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:08,348 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:08,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:08,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:08,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:08,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:08,349 DEBUG [Thread-2971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:08,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40116 deadline: 1682193428340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,349 DEBUG [Thread-2971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:08,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:08,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40156 deadline: 1682193428341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:08,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:08,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1682193428352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:08,414 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/487c7ac2c56147a0ac8d4557b00af58b 2023-04-22 19:56:08,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/17fcb5c39a784c59b797765fc5040a53 2023-04-22 19:56:08,425 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/155f313a703d43bba94306451d6c5ddd as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/155f313a703d43bba94306451d6c5ddd 2023-04-22 19:56:08,428 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/155f313a703d43bba94306451d6c5ddd, entries=200, sequenceid=538, filesize=14.3 K 2023-04-22 19:56:08,429 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/487c7ac2c56147a0ac8d4557b00af58b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/487c7ac2c56147a0ac8d4557b00af58b 2023-04-22 19:56:08,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/487c7ac2c56147a0ac8d4557b00af58b, entries=150, sequenceid=538, filesize=11.9 K 2023-04-22 19:56:08,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/17fcb5c39a784c59b797765fc5040a53 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/17fcb5c39a784c59b797765fc5040a53 2023-04-22 19:56:08,436 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/17fcb5c39a784c59b797765fc5040a53, entries=150, sequenceid=538, filesize=11.9 K 2023-04-22 19:56:08,437 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 62116471b8dabdbd8f57094f112778d1 in 441ms, sequenceid=538, compaction requested=true 2023-04-22 19:56:08,437 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:08,437 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:08,437 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:08,437 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-22 19:56:08,437 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-22 19:56:08,437 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:08,440 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 110356 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-22 19:56:08,440 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 112796 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-22 19:56:08,441 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/B is initiating minor compaction (all files) 2023-04-22 19:56:08,441 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/A is initiating minor compaction (all files) 2023-04-22 19:56:08,441 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/B in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:08,441 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/A in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:08,441 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/f2e639070ba140ed9006da2b2add255a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9a97aad1893e43d2a3da0d8b4dcd56f5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fbf87316beb74cc2908cf7db6fa57948, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/85d4811681604799aaa6944f7d217b0a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/4ac5d536eca94aeeb65402ceec9ef473, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/c7c113a7de424bc681089b1cb4e4c4c0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1457d564d7af44e58ab1b8db70c4ea70, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/b563ed304dfd4b5fb3d450b56ed0d809, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/487c7ac2c56147a0ac8d4557b00af58b] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=107.8 K 2023-04-22 19:56:08,441 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/ab0b0631a4af47f78eafea675a52712b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d72ed41aed5542228bfac85c6f0cbcb2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/78d4c47c07b24272b6d4be4f08d69d5b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/c3c02928892e4f7a90194d0da54c2c63, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/83ddd8eba0e84e1bbfa1c0417c582196, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/fc6ea7ce06044420a03e2733b1b25d5b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/cf27704d56344b6f94a3ee16857c2e38, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/47c46c0059e44e71a0d5b5771ddee4da, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/155f313a703d43bba94306451d6c5ddd] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=110.2 K 2023-04-22 19:56:08,441 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f2e639070ba140ed9006da2b2add255a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682193362235 2023-04-22 19:56:08,441 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting ab0b0631a4af47f78eafea675a52712b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682193362235 2023-04-22 19:56:08,442 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 9a97aad1893e43d2a3da0d8b4dcd56f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682193362273 2023-04-22 19:56:08,442 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting d72ed41aed5542228bfac85c6f0cbcb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682193362273 2023-04-22 19:56:08,442 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting fbf87316beb74cc2908cf7db6fa57948, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1682193362471 2023-04-22 19:56:08,442 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 78d4c47c07b24272b6d4be4f08d69d5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1682193362471 2023-04-22 19:56:08,442 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 85d4811681604799aaa6944f7d217b0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1682193363125 2023-04-22 19:56:08,443 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c3c02928892e4f7a90194d0da54c2c63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1682193363125 2023-04-22 19:56:08,443 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 4ac5d536eca94aeeb65402ceec9ef473, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1682193364320 2023-04-22 19:56:08,443 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 83ddd8eba0e84e1bbfa1c0417c582196, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1682193364320 2023-04-22 19:56:08,443 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting c7c113a7de424bc681089b1cb4e4c4c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682193364646 2023-04-22 19:56:08,443 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting fc6ea7ce06044420a03e2733b1b25d5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682193364646 2023-04-22 19:56:08,443 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1457d564d7af44e58ab1b8db70c4ea70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1682193365311 2023-04-22 19:56:08,444 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting cf27704d56344b6f94a3ee16857c2e38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1682193365311 2023-04-22 19:56:08,444 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b563ed304dfd4b5fb3d450b56ed0d809, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682193366468 2023-04-22 19:56:08,444 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 47c46c0059e44e71a0d5b5771ddee4da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682193366468 2023-04-22 19:56:08,444 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 487c7ac2c56147a0ac8d4557b00af58b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1682193367666 2023-04-22 19:56:08,444 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 155f313a703d43bba94306451d6c5ddd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1682193367666 2023-04-22 19:56:08,446 DEBUG [Thread-2972] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:53381 2023-04-22 19:56:08,447 DEBUG [Thread-2972] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:08,449 DEBUG [Thread-2974] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:53381 2023-04-22 19:56:08,449 DEBUG [Thread-2974] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:08,449 DEBUG [Thread-2971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:08,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:08,451 DEBUG [Thread-2980] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x211582f2 to 127.0.0.1:53381 2023-04-22 19:56:08,451 DEBUG [Thread-2980] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:08,456 DEBUG [Thread-2978] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f5363d8 to 127.0.0.1:53381 2023-04-22 19:56:08,456 DEBUG [Thread-2978] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:08,458 DEBUG [Thread-2976] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:53381 2023-04-22 19:56:08,458 DEBUG [Thread-2976] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:08,459 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#A#compaction#470 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:08,460 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#B#compaction#471 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:08,475 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/866196969fc64071941a714a3fe2de6a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/866196969fc64071941a714a3fe2de6a 2023-04-22 19:56:08,479 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/B of 62116471b8dabdbd8f57094f112778d1 into 866196969fc64071941a714a3fe2de6a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:08,479 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:08,479 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/B, priority=7, startTime=1682193368437; duration=0sec 2023-04-22 19:56:08,479 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:08,479 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-22 19:56:08,481 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 110356 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-22 19:56:08,481 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): 62116471b8dabdbd8f57094f112778d1/C is initiating minor compaction (all files) 2023-04-22 19:56:08,481 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 62116471b8dabdbd8f57094f112778d1/C in TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:08,481 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/7f08091a2899460b9e058318ef4fc3f2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/198764ff047048bea529b7849293da20, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6a9887f2b7042848ef3cdd47d946fe9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/0b718003ce1b461e90da1dbab1fde5ea, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6db5840cb8948a79c50bc8a9fa7574e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/13e5ae975d2d4c4e847595e52ab2a733, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/de2fada3043e48f7bbc049cbd3dcdd93, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/a4548e37f37a4fee8f341ab8ea7b4e74, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/17fcb5c39a784c59b797765fc5040a53] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp, totalSize=107.8 K 2023-04-22 19:56:08,481 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 7f08091a2899460b9e058318ef4fc3f2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682193362235 2023-04-22 19:56:08,482 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 198764ff047048bea529b7849293da20, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682193362273 2023-04-22 19:56:08,482 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f6a9887f2b7042848ef3cdd47d946fe9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1682193362471 2023-04-22 19:56:08,482 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0b718003ce1b461e90da1dbab1fde5ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1682193363125 2023-04-22 19:56:08,482 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f6db5840cb8948a79c50bc8a9fa7574e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1682193364320 2023-04-22 19:56:08,482 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 13e5ae975d2d4c4e847595e52ab2a733, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682193364646 2023-04-22 19:56:08,483 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting de2fada3043e48f7bbc049cbd3dcdd93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1682193365311 2023-04-22 19:56:08,483 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a4548e37f37a4fee8f341ab8ea7b4e74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682193366468 2023-04-22 19:56:08,483 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 17fcb5c39a784c59b797765fc5040a53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1682193367666 2023-04-22 19:56:08,492 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): 62116471b8dabdbd8f57094f112778d1#C#compaction#472 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:08,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:08,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 19:56:08,653 DEBUG [Thread-2963] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x501c5a88 to 127.0.0.1:53381 2023-04-22 19:56:08,654 DEBUG [Thread-2963] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:08,654 DEBUG [Thread-2969] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c29757d to 127.0.0.1:53381 2023-04-22 19:56:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:08,654 DEBUG [Thread-2969] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:08,668 DEBUG [Thread-2967] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34ad4aa8 to 127.0.0.1:53381 2023-04-22 19:56:08,668 DEBUG [Thread-2967] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:08,866 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/0cc657438134489f81fd3d7dffcc920e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0cc657438134489f81fd3d7dffcc920e 2023-04-22 19:56:08,870 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/A of 62116471b8dabdbd8f57094f112778d1 into 0cc657438134489f81fd3d7dffcc920e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:08,870 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:08,870 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/A, priority=7, startTime=1682193368437; duration=0sec 2023-04-22 19:56:08,870 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:08,900 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/809fd9c4c86440ed932035d7b5e524a1 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/809fd9c4c86440ed932035d7b5e524a1 2023-04-22 19:56:08,903 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 62116471b8dabdbd8f57094f112778d1/C of 62116471b8dabdbd8f57094f112778d1 into 809fd9c4c86440ed932035d7b5e524a1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:08,903 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:08,903 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1., storeName=62116471b8dabdbd8f57094f112778d1/C, priority=7, startTime=1682193368437; duration=0sec 2023-04-22 19:56:08,903 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:09,060 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/b9124f39272245339d18079f5a62b015 2023-04-22 19:56:09,469 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/5cd89001ebb74d1daabdfc19ad33f406 2023-04-22 19:56:09,878 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/b1e454e3b1904d3f91d0b48a9dfed19d 2023-04-22 19:56:09,881 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/b9124f39272245339d18079f5a62b015 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/b9124f39272245339d18079f5a62b015 2023-04-22 19:56:09,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/b9124f39272245339d18079f5a62b015, entries=150, sequenceid=552, filesize=11.9 K 2023-04-22 19:56:09,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/5cd89001ebb74d1daabdfc19ad33f406 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/5cd89001ebb74d1daabdfc19ad33f406 2023-04-22 19:56:09,887 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/5cd89001ebb74d1daabdfc19ad33f406, entries=150, sequenceid=552, filesize=11.9 K 2023-04-22 19:56:09,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/b1e454e3b1904d3f91d0b48a9dfed19d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/b1e454e3b1904d3f91d0b48a9dfed19d 2023-04-22 19:56:09,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/b1e454e3b1904d3f91d0b48a9dfed19d, entries=150, sequenceid=552, filesize=11.9 K 2023-04-22 19:56:09,891 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=6.71 KB/6870 for 62116471b8dabdbd8f57094f112778d1 in 1238ms, sequenceid=552, compaction requested=false 2023-04-22 19:56:09,891 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:14,380 DEBUG [Thread-2965] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27f2d997 to 127.0.0.1:53381 2023-04-22 19:56:14,380 DEBUG [Thread-2965] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:16,763 DEBUG [Thread-2961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e372785 to 127.0.0.1:53381 2023-04-22 19:56:16,763 DEBUG [Thread-2961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 18 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 55 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 149 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 165 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1670 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 5010 rows 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1667 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 5000 rows 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1668 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 4989 rows 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1678 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 5025 rows 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(402): scanned 1660 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(403): verified 4980 rows 2023-04-22 19:56:16,764 INFO [Listener at localhost/46665] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 19:56:16,765 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a29ce5a to 127.0.0.1:53381 2023-04-22 19:56:16,765 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:16,766 INFO [Listener at localhost/46665] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 19:56:16,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 19:56:16,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:16,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 19:56:16,771 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193376771"}]},"ts":"1682193376771"} 2023-04-22 19:56:16,772 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 19:56:16,774 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 19:56:16,775 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=62116471b8dabdbd8f57094f112778d1, UNASSIGN}] 2023-04-22 19:56:16,777 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=62116471b8dabdbd8f57094f112778d1, UNASSIGN 2023-04-22 19:56:16,777 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=62116471b8dabdbd8f57094f112778d1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:16,777 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193376777"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193376777"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193376777"}]},"ts":"1682193376777"} 2023-04-22 19:56:16,778 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:56:16,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 19:56:16,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:16,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 62116471b8dabdbd8f57094f112778d1, disabling compactions & flushes 2023-04-22 19:56:16,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:16,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:16,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. after waiting 0 ms 2023-04-22 19:56:16,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:16,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 62116471b8dabdbd8f57094f112778d1 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-22 19:56:16,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=A 2023-04-22 19:56:16,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:16,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=B 2023-04-22 19:56:16,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:16,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 62116471b8dabdbd8f57094f112778d1, store=C 2023-04-22 19:56:16,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:17,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 19:56:17,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/49c607575cda4ef58478448d9ed2796d 2023-04-22 19:56:17,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 19:56:17,746 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/181a0faac88d43a8a6df3e350444db28 2023-04-22 19:56:17,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 19:56:18,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/ecab426256c349d689127aaa5e87b63b 2023-04-22 19:56:18,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/A/49c607575cda4ef58478448d9ed2796d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/49c607575cda4ef58478448d9ed2796d 2023-04-22 19:56:18,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/49c607575cda4ef58478448d9ed2796d, entries=150, sequenceid=560, filesize=11.9 K 2023-04-22 19:56:18,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/B/181a0faac88d43a8a6df3e350444db28 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/181a0faac88d43a8a6df3e350444db28 2023-04-22 19:56:18,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/181a0faac88d43a8a6df3e350444db28, entries=150, sequenceid=560, filesize=11.9 K 2023-04-22 19:56:18,165 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/.tmp/C/ecab426256c349d689127aaa5e87b63b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ecab426256c349d689127aaa5e87b63b 2023-04-22 19:56:18,167 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ecab426256c349d689127aaa5e87b63b, entries=150, sequenceid=560, filesize=11.9 K 2023-04-22 19:56:18,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 62116471b8dabdbd8f57094f112778d1 in 1238ms, sequenceid=560, compaction requested=true 2023-04-22 19:56:18,169 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9660a007f5644680a1be7a919b852261, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/93edf50fbce241c0b99c82b4c660ab12, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9236676d203c44f2a95bd2162e667fa9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/8846415461d44d1ba45ded2619946ea2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2d5f56e2d6f34a549e77080cc845a7e6, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2e472179723f4aa0803da70c4299466f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/03370b7bee704702b05777c9cf566bfc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/e29cb332d64d47d48da9aeff65e58524, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/ab3fd162c27648908b1b37a896087ada, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/69740598d7a243678ef3983c9a16cf86, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0083291a33a94f5da99174c4e454c9ad, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/e3777abf89444a369bd93b3ece4bbb5b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/454e370bdea640a7b741cf9bb8ad6cbf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/a6939a98f1914b1e8983d5561539b61a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/fc709c3b6fe44362bff2012dab0618de, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/b32ecd9ac2694168a18da42b67b5e882, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0b50bc19929b4b2aaa250c7456d36b54, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/60f9a2db08d5454fbe1adb9129a61160, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d5ed5df59c424df1935b76ef2e17cafb, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/5e95fbf5b6f04660a85060bb4dffcf10, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/ab0b0631a4af47f78eafea675a52712b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d72ed41aed5542228bfac85c6f0cbcb2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/78d4c47c07b24272b6d4be4f08d69d5b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/c3c02928892e4f7a90194d0da54c2c63, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/83ddd8eba0e84e1bbfa1c0417c582196, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/fc6ea7ce06044420a03e2733b1b25d5b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/cf27704d56344b6f94a3ee16857c2e38, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/47c46c0059e44e71a0d5b5771ddee4da, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/155f313a703d43bba94306451d6c5ddd] to archive 2023-04-22 19:56:18,170 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:56:18,171 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9660a007f5644680a1be7a919b852261 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9660a007f5644680a1be7a919b852261 2023-04-22 19:56:18,172 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/93edf50fbce241c0b99c82b4c660ab12 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/93edf50fbce241c0b99c82b4c660ab12 2023-04-22 19:56:18,173 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9236676d203c44f2a95bd2162e667fa9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/9236676d203c44f2a95bd2162e667fa9 2023-04-22 19:56:18,174 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/8846415461d44d1ba45ded2619946ea2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/8846415461d44d1ba45ded2619946ea2 2023-04-22 19:56:18,175 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2d5f56e2d6f34a549e77080cc845a7e6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2d5f56e2d6f34a549e77080cc845a7e6 2023-04-22 19:56:18,176 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2e472179723f4aa0803da70c4299466f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/2e472179723f4aa0803da70c4299466f 2023-04-22 19:56:18,177 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/03370b7bee704702b05777c9cf566bfc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/03370b7bee704702b05777c9cf566bfc 2023-04-22 19:56:18,178 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/e29cb332d64d47d48da9aeff65e58524 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/e29cb332d64d47d48da9aeff65e58524 2023-04-22 19:56:18,179 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/ab3fd162c27648908b1b37a896087ada to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/ab3fd162c27648908b1b37a896087ada 2023-04-22 19:56:18,180 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/69740598d7a243678ef3983c9a16cf86 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/69740598d7a243678ef3983c9a16cf86 2023-04-22 19:56:18,181 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0083291a33a94f5da99174c4e454c9ad to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0083291a33a94f5da99174c4e454c9ad 2023-04-22 19:56:18,182 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/e3777abf89444a369bd93b3ece4bbb5b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/e3777abf89444a369bd93b3ece4bbb5b 2023-04-22 19:56:18,182 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/454e370bdea640a7b741cf9bb8ad6cbf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/454e370bdea640a7b741cf9bb8ad6cbf 2023-04-22 19:56:18,183 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/a6939a98f1914b1e8983d5561539b61a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/a6939a98f1914b1e8983d5561539b61a 2023-04-22 19:56:18,184 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/fc709c3b6fe44362bff2012dab0618de to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/fc709c3b6fe44362bff2012dab0618de 2023-04-22 19:56:18,184 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/b32ecd9ac2694168a18da42b67b5e882 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/b32ecd9ac2694168a18da42b67b5e882 2023-04-22 19:56:18,185 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0b50bc19929b4b2aaa250c7456d36b54 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0b50bc19929b4b2aaa250c7456d36b54 2023-04-22 19:56:18,186 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/60f9a2db08d5454fbe1adb9129a61160 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/60f9a2db08d5454fbe1adb9129a61160 2023-04-22 19:56:18,186 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d5ed5df59c424df1935b76ef2e17cafb to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d5ed5df59c424df1935b76ef2e17cafb 2023-04-22 19:56:18,187 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/5e95fbf5b6f04660a85060bb4dffcf10 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/5e95fbf5b6f04660a85060bb4dffcf10 2023-04-22 19:56:18,188 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/ab0b0631a4af47f78eafea675a52712b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/ab0b0631a4af47f78eafea675a52712b 2023-04-22 19:56:18,189 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d72ed41aed5542228bfac85c6f0cbcb2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/d72ed41aed5542228bfac85c6f0cbcb2 2023-04-22 19:56:18,190 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/78d4c47c07b24272b6d4be4f08d69d5b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/78d4c47c07b24272b6d4be4f08d69d5b 2023-04-22 19:56:18,190 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/c3c02928892e4f7a90194d0da54c2c63 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/c3c02928892e4f7a90194d0da54c2c63 2023-04-22 19:56:18,191 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/83ddd8eba0e84e1bbfa1c0417c582196 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/83ddd8eba0e84e1bbfa1c0417c582196 2023-04-22 19:56:18,192 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/fc6ea7ce06044420a03e2733b1b25d5b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/fc6ea7ce06044420a03e2733b1b25d5b 2023-04-22 19:56:18,193 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/cf27704d56344b6f94a3ee16857c2e38 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/cf27704d56344b6f94a3ee16857c2e38 2023-04-22 19:56:18,193 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/47c46c0059e44e71a0d5b5771ddee4da to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/47c46c0059e44e71a0d5b5771ddee4da 2023-04-22 19:56:18,194 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/155f313a703d43bba94306451d6c5ddd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/155f313a703d43bba94306451d6c5ddd 2023-04-22 19:56:18,197 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/5e019f4a0a6f44cbb3c9952cda27e701, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1ce6a0aa20f443819689b2dd92ff625c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/e257c1d05d6f4295bba16138e04e0051, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/6aa4f8ef15d74e999eb061ea86038a20, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bd6b5abfac434e00bf08be959bf6f6d2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bbbccc995cb541999e60f9b2a66f2116, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/80acd121bac3484da29d3b7c1487c794, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/a8452519ef0d4efd8f60127a335417de, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/7be696c57e46452a87447b42041d0a67, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/597714bd4c3b43c193ff33199a550a61, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/3bd339a3f307437bbb71e956abbad25c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/8f68e83319624aeb9071d7acdedc35b4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/dd6e1bf4c64547d1ba659fce76796c69, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/907617583a4848849953227eb740d1a2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9686781a1db841bdaf2366dc5f670888, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bea51f271e214027bab2eb949b21ef60, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/db7c5e2092794131abc24ceb9314a721, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/26d4c85f8b6a432084bcc847877c114e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fde7f7107df5447c89902a4ce45f81fd, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/f2e639070ba140ed9006da2b2add255a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/807eb5214f344c5f8d928c1cf54b113f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9a97aad1893e43d2a3da0d8b4dcd56f5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fbf87316beb74cc2908cf7db6fa57948, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/85d4811681604799aaa6944f7d217b0a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/4ac5d536eca94aeeb65402ceec9ef473, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/c7c113a7de424bc681089b1cb4e4c4c0, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1457d564d7af44e58ab1b8db70c4ea70, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/b563ed304dfd4b5fb3d450b56ed0d809, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/487c7ac2c56147a0ac8d4557b00af58b] to archive 2023-04-22 19:56:18,197 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:56:18,199 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/5e019f4a0a6f44cbb3c9952cda27e701 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/5e019f4a0a6f44cbb3c9952cda27e701 2023-04-22 19:56:18,200 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1ce6a0aa20f443819689b2dd92ff625c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1ce6a0aa20f443819689b2dd92ff625c 2023-04-22 19:56:18,201 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/e257c1d05d6f4295bba16138e04e0051 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/e257c1d05d6f4295bba16138e04e0051 2023-04-22 19:56:18,202 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/6aa4f8ef15d74e999eb061ea86038a20 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/6aa4f8ef15d74e999eb061ea86038a20 2023-04-22 19:56:18,203 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bd6b5abfac434e00bf08be959bf6f6d2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bd6b5abfac434e00bf08be959bf6f6d2 2023-04-22 19:56:18,204 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bbbccc995cb541999e60f9b2a66f2116 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bbbccc995cb541999e60f9b2a66f2116 2023-04-22 19:56:18,205 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/80acd121bac3484da29d3b7c1487c794 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/80acd121bac3484da29d3b7c1487c794 2023-04-22 19:56:18,206 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/a8452519ef0d4efd8f60127a335417de to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/a8452519ef0d4efd8f60127a335417de 2023-04-22 19:56:18,206 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/7be696c57e46452a87447b42041d0a67 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/7be696c57e46452a87447b42041d0a67 2023-04-22 19:56:18,207 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/597714bd4c3b43c193ff33199a550a61 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/597714bd4c3b43c193ff33199a550a61 2023-04-22 19:56:18,208 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/3bd339a3f307437bbb71e956abbad25c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/3bd339a3f307437bbb71e956abbad25c 2023-04-22 19:56:18,209 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/8f68e83319624aeb9071d7acdedc35b4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/8f68e83319624aeb9071d7acdedc35b4 2023-04-22 19:56:18,210 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/dd6e1bf4c64547d1ba659fce76796c69 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/dd6e1bf4c64547d1ba659fce76796c69 2023-04-22 19:56:18,211 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/907617583a4848849953227eb740d1a2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/907617583a4848849953227eb740d1a2 2023-04-22 19:56:18,212 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9686781a1db841bdaf2366dc5f670888 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9686781a1db841bdaf2366dc5f670888 2023-04-22 19:56:18,212 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bea51f271e214027bab2eb949b21ef60 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/bea51f271e214027bab2eb949b21ef60 2023-04-22 19:56:18,213 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/db7c5e2092794131abc24ceb9314a721 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/db7c5e2092794131abc24ceb9314a721 2023-04-22 19:56:18,214 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/26d4c85f8b6a432084bcc847877c114e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/26d4c85f8b6a432084bcc847877c114e 2023-04-22 19:56:18,216 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fde7f7107df5447c89902a4ce45f81fd to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fde7f7107df5447c89902a4ce45f81fd 2023-04-22 19:56:18,217 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/f2e639070ba140ed9006da2b2add255a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/f2e639070ba140ed9006da2b2add255a 2023-04-22 19:56:18,217 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/807eb5214f344c5f8d928c1cf54b113f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/807eb5214f344c5f8d928c1cf54b113f 2023-04-22 19:56:18,218 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9a97aad1893e43d2a3da0d8b4dcd56f5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/9a97aad1893e43d2a3da0d8b4dcd56f5 2023-04-22 19:56:18,219 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fbf87316beb74cc2908cf7db6fa57948 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/fbf87316beb74cc2908cf7db6fa57948 2023-04-22 19:56:18,220 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/85d4811681604799aaa6944f7d217b0a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/85d4811681604799aaa6944f7d217b0a 2023-04-22 19:56:18,220 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/4ac5d536eca94aeeb65402ceec9ef473 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/4ac5d536eca94aeeb65402ceec9ef473 2023-04-22 19:56:18,221 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/c7c113a7de424bc681089b1cb4e4c4c0 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/c7c113a7de424bc681089b1cb4e4c4c0 2023-04-22 19:56:18,222 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1457d564d7af44e58ab1b8db70c4ea70 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/1457d564d7af44e58ab1b8db70c4ea70 2023-04-22 19:56:18,222 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/b563ed304dfd4b5fb3d450b56ed0d809 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/b563ed304dfd4b5fb3d450b56ed0d809 2023-04-22 19:56:18,223 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/487c7ac2c56147a0ac8d4557b00af58b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/487c7ac2c56147a0ac8d4557b00af58b 2023-04-22 19:56:18,226 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ef7d9ab3d9e3405fa4fa94d244ac41db, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/812604ed7d514be6a5a75259b298de1b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/c8887bb66e714b3b9911c3771bdb2caf, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/1f0a46d20a7444cab37eb691df7dbd93, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d5bc1cfb6b4a4379a73411180f1c723c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ebb268245b824ab6b3943a744ecdefa5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/7cc0b5a8d134400f9f829fa8854d6b25, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/67deee18b02e49769f9d9b2868fc62b5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/68f26ce9bff8473d8e46f13c41af89a8, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5c2ea1267721436bb457a2b756ddc44e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/0f07b9f7e546492399039ba636704c8e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5a99ba7afb984df58f30f8511553cb98, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d415b7eee5bb4b0a8f3358bd9e5b4e44, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5c2577c870d541279e0bb6e80846f352, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/8508adc92a9142c781bab23e559e0cb3, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/bf8e2a1fc63a4dd889b8352192fa87bc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/796fd1c9b2424f2b8cf87b66571efcfe, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/af92df713b6b41daa0cb345c33effa2a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d7bfa6e5327d4ee19d34bc8009238e36, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/7f08091a2899460b9e058318ef4fc3f2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/b55547c342e449f99652a018ff2d159f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/198764ff047048bea529b7849293da20, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6a9887f2b7042848ef3cdd47d946fe9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/0b718003ce1b461e90da1dbab1fde5ea, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6db5840cb8948a79c50bc8a9fa7574e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/13e5ae975d2d4c4e847595e52ab2a733, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/de2fada3043e48f7bbc049cbd3dcdd93, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/a4548e37f37a4fee8f341ab8ea7b4e74, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/17fcb5c39a784c59b797765fc5040a53] to archive 2023-04-22 19:56:18,227 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:56:18,228 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ef7d9ab3d9e3405fa4fa94d244ac41db to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ef7d9ab3d9e3405fa4fa94d244ac41db 2023-04-22 19:56:18,229 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/812604ed7d514be6a5a75259b298de1b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/812604ed7d514be6a5a75259b298de1b 2023-04-22 19:56:18,230 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/c8887bb66e714b3b9911c3771bdb2caf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/c8887bb66e714b3b9911c3771bdb2caf 2023-04-22 19:56:18,231 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/1f0a46d20a7444cab37eb691df7dbd93 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/1f0a46d20a7444cab37eb691df7dbd93 2023-04-22 19:56:18,231 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d5bc1cfb6b4a4379a73411180f1c723c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d5bc1cfb6b4a4379a73411180f1c723c 2023-04-22 19:56:18,232 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ebb268245b824ab6b3943a744ecdefa5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ebb268245b824ab6b3943a744ecdefa5 2023-04-22 19:56:18,233 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/7cc0b5a8d134400f9f829fa8854d6b25 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/7cc0b5a8d134400f9f829fa8854d6b25 2023-04-22 19:56:18,234 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/67deee18b02e49769f9d9b2868fc62b5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/67deee18b02e49769f9d9b2868fc62b5 2023-04-22 19:56:18,234 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/68f26ce9bff8473d8e46f13c41af89a8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/68f26ce9bff8473d8e46f13c41af89a8 2023-04-22 19:56:18,235 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5c2ea1267721436bb457a2b756ddc44e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5c2ea1267721436bb457a2b756ddc44e 2023-04-22 19:56:18,236 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/0f07b9f7e546492399039ba636704c8e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/0f07b9f7e546492399039ba636704c8e 2023-04-22 19:56:18,237 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5a99ba7afb984df58f30f8511553cb98 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5a99ba7afb984df58f30f8511553cb98 2023-04-22 19:56:18,237 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d415b7eee5bb4b0a8f3358bd9e5b4e44 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d415b7eee5bb4b0a8f3358bd9e5b4e44 2023-04-22 19:56:18,238 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5c2577c870d541279e0bb6e80846f352 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/5c2577c870d541279e0bb6e80846f352 2023-04-22 19:56:18,239 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/8508adc92a9142c781bab23e559e0cb3 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/8508adc92a9142c781bab23e559e0cb3 2023-04-22 19:56:18,240 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/bf8e2a1fc63a4dd889b8352192fa87bc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/bf8e2a1fc63a4dd889b8352192fa87bc 2023-04-22 19:56:18,241 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/796fd1c9b2424f2b8cf87b66571efcfe to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/796fd1c9b2424f2b8cf87b66571efcfe 2023-04-22 19:56:18,241 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/af92df713b6b41daa0cb345c33effa2a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/af92df713b6b41daa0cb345c33effa2a 2023-04-22 19:56:18,242 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d7bfa6e5327d4ee19d34bc8009238e36 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/d7bfa6e5327d4ee19d34bc8009238e36 2023-04-22 19:56:18,243 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/7f08091a2899460b9e058318ef4fc3f2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/7f08091a2899460b9e058318ef4fc3f2 2023-04-22 19:56:18,243 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/b55547c342e449f99652a018ff2d159f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/b55547c342e449f99652a018ff2d159f 2023-04-22 19:56:18,244 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/198764ff047048bea529b7849293da20 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/198764ff047048bea529b7849293da20 2023-04-22 19:56:18,245 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6a9887f2b7042848ef3cdd47d946fe9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6a9887f2b7042848ef3cdd47d946fe9 2023-04-22 19:56:18,246 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/0b718003ce1b461e90da1dbab1fde5ea to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/0b718003ce1b461e90da1dbab1fde5ea 2023-04-22 19:56:18,247 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6db5840cb8948a79c50bc8a9fa7574e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/f6db5840cb8948a79c50bc8a9fa7574e 2023-04-22 19:56:18,248 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/13e5ae975d2d4c4e847595e52ab2a733 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/13e5ae975d2d4c4e847595e52ab2a733 2023-04-22 19:56:18,248 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/de2fada3043e48f7bbc049cbd3dcdd93 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/de2fada3043e48f7bbc049cbd3dcdd93 2023-04-22 19:56:18,249 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/a4548e37f37a4fee8f341ab8ea7b4e74 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/a4548e37f37a4fee8f341ab8ea7b4e74 2023-04-22 19:56:18,250 DEBUG [StoreCloser-TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/17fcb5c39a784c59b797765fc5040a53 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/17fcb5c39a784c59b797765fc5040a53 2023-04-22 19:56:18,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/recovered.edits/563.seqid, newMaxSeqId=563, maxSeqId=1 2023-04-22 19:56:18,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1. 2023-04-22 19:56:18,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 62116471b8dabdbd8f57094f112778d1: 2023-04-22 19:56:18,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:18,258 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=62116471b8dabdbd8f57094f112778d1, regionState=CLOSED 2023-04-22 19:56:18,258 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193378258"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193378258"}]},"ts":"1682193378258"} 2023-04-22 19:56:18,260 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-22 19:56:18,260 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 62116471b8dabdbd8f57094f112778d1, server=jenkins-hbase4.apache.org,44791,1682193215766 in 1.4810 sec 2023-04-22 19:56:18,261 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-22 19:56:18,261 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=62116471b8dabdbd8f57094f112778d1, UNASSIGN in 1.4850 sec 2023-04-22 19:56:18,262 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193378262"}]},"ts":"1682193378262"} 2023-04-22 19:56:18,262 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 19:56:18,264 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 19:56:18,265 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4970 sec 2023-04-22 19:56:18,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 19:56:18,875 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-22 19:56:18,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 19:56:18,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:18,878 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:18,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-22 19:56:18,879 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:18,885 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:18,887 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/recovered.edits] 2023-04-22 19:56:18,889 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0cc657438134489f81fd3d7dffcc920e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/0cc657438134489f81fd3d7dffcc920e 2023-04-22 19:56:18,890 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/49c607575cda4ef58478448d9ed2796d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/49c607575cda4ef58478448d9ed2796d 2023-04-22 19:56:18,891 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/b9124f39272245339d18079f5a62b015 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/A/b9124f39272245339d18079f5a62b015 2023-04-22 19:56:18,893 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/181a0faac88d43a8a6df3e350444db28 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/181a0faac88d43a8a6df3e350444db28 2023-04-22 19:56:18,894 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/5cd89001ebb74d1daabdfc19ad33f406 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/5cd89001ebb74d1daabdfc19ad33f406 2023-04-22 19:56:18,895 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/866196969fc64071941a714a3fe2de6a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/B/866196969fc64071941a714a3fe2de6a 2023-04-22 19:56:18,897 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/809fd9c4c86440ed932035d7b5e524a1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/809fd9c4c86440ed932035d7b5e524a1 2023-04-22 19:56:18,898 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/b1e454e3b1904d3f91d0b48a9dfed19d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/b1e454e3b1904d3f91d0b48a9dfed19d 2023-04-22 19:56:18,898 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ecab426256c349d689127aaa5e87b63b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/C/ecab426256c349d689127aaa5e87b63b 2023-04-22 19:56:18,900 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/recovered.edits/563.seqid to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1/recovered.edits/563.seqid 2023-04-22 19:56:18,901 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/62116471b8dabdbd8f57094f112778d1 2023-04-22 19:56:18,901 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:56:18,903 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:18,904 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 19:56:18,905 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 19:56:18,906 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:18,906 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 19:56:18,906 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682193378906"}]},"ts":"9223372036854775807"} 2023-04-22 19:56:18,907 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 19:56:18,907 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 62116471b8dabdbd8f57094f112778d1, NAME => 'TestAcidGuarantees,,1682193346257.62116471b8dabdbd8f57094f112778d1.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 19:56:18,907 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 19:56:18,908 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682193378908"}]},"ts":"9223372036854775807"} 2023-04-22 19:56:18,909 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 19:56:18,910 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:18,911 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 34 msec 2023-04-22 19:56:18,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-22 19:56:18,980 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-22 19:56:18,992 INFO [Listener at localhost/46665] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 230), OpenFileDescriptor=524 (was 542), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=397 (was 413), ProcessCount=170 (was 170), AvailableMemoryMB=12850 (was 12727) - AvailableMemoryMB LEAK? - 2023-04-22 19:56:19,008 INFO [Listener at localhost/46665] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=524, MaxFileDescriptor=60000, SystemLoadAverage=397, ProcessCount=170, AvailableMemoryMB=12849 2023-04-22 19:56:19,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 19:56:19,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 19:56:19,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:19,012 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 19:56:19,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-22 19:56:19,013 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 19:56:19,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 19:56:19,014 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:19,014 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261 empty. 2023-04-22 19:56:19,015 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:19,015 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:56:19,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 19:56:19,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 19:56:19,427 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 19:56:19,428 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e8d85766c3fe2800a849fb8577794261, NAME => 'TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp 2023-04-22 19:56:19,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 19:56:19,833 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:56:19,833 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e8d85766c3fe2800a849fb8577794261, disabling compactions & flushes 2023-04-22 19:56:19,833 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:19,834 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:19,834 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. after waiting 0 ms 2023-04-22 19:56:19,834 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:19,834 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:19,834 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:19,836 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 19:56:19,836 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193379836"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193379836"}]},"ts":"1682193379836"} 2023-04-22 19:56:19,837 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 19:56:19,838 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 19:56:19,838 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193379838"}]},"ts":"1682193379838"} 2023-04-22 19:56:19,839 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 19:56:19,845 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8d85766c3fe2800a849fb8577794261, ASSIGN}] 2023-04-22 19:56:19,846 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8d85766c3fe2800a849fb8577794261, ASSIGN 2023-04-22 19:56:19,847 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8d85766c3fe2800a849fb8577794261, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44791,1682193215766; forceNewPlan=false, retain=false 2023-04-22 19:56:19,998 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e8d85766c3fe2800a849fb8577794261, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:19,998 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193379998"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193379998"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193379998"}]},"ts":"1682193379998"} 2023-04-22 19:56:20,000 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:56:20,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 19:56:20,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:20,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e8d85766c3fe2800a849fb8577794261, NAME => 'TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:56:20,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:20,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:56:20,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:20,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:20,156 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:20,157 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A 2023-04-22 19:56:20,157 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A 2023-04-22 19:56:20,158 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:56:20,158 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8d85766c3fe2800a849fb8577794261 columnFamilyName A 2023-04-22 19:56:20,158 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(310): Store=e8d85766c3fe2800a849fb8577794261/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:56:20,158 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:20,159 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B 2023-04-22 19:56:20,159 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B 2023-04-22 19:56:20,159 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:56:20,160 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8d85766c3fe2800a849fb8577794261 columnFamilyName B 2023-04-22 19:56:20,160 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(310): Store=e8d85766c3fe2800a849fb8577794261/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:56:20,160 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:20,161 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C 2023-04-22 19:56:20,161 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C 2023-04-22 19:56:20,161 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:56:20,161 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8d85766c3fe2800a849fb8577794261 columnFamilyName C 2023-04-22 19:56:20,162 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(310): Store=e8d85766c3fe2800a849fb8577794261/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:56:20,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:20,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:20,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:20,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 19:56:20,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:20,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 19:56:20,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e8d85766c3fe2800a849fb8577794261; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72947732, jitterRate=0.08700591325759888}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 19:56:20,168 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:20,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., pid=56, masterSystemTime=1682193380151 2023-04-22 19:56:20,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:20,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:20,170 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e8d85766c3fe2800a849fb8577794261, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:20,170 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193380170"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682193380170"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682193380170"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682193380170"}]},"ts":"1682193380170"} 2023-04-22 19:56:20,172 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-22 19:56:20,172 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 in 171 msec 2023-04-22 19:56:20,173 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-22 19:56:20,173 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8d85766c3fe2800a849fb8577794261, ASSIGN in 327 msec 2023-04-22 19:56:20,174 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 19:56:20,174 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193380174"}]},"ts":"1682193380174"} 2023-04-22 19:56:20,175 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 19:56:20,176 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 19:56:20,177 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1670 sec 2023-04-22 19:56:21,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 19:56:21,118 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-22 19:56:21,118 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x3011deb3 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:56:21,123 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b480bc5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:56:21,124 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:56:21,126 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:56:21,126 DEBUG [Listener at localhost/46665] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 19:56:21,129 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35074, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 19:56:21,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 19:56:21,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 19:56:21,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:21,540 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 19:56:21,541 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 19:56:21,541 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 19:56:21,543 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-22 19:56:21,545 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8d85766c3fe2800a849fb8577794261, REOPEN/MOVE}] 2023-04-22 19:56:21,546 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8d85766c3fe2800a849fb8577794261, REOPEN/MOVE 2023-04-22 19:56:21,546 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e8d85766c3fe2800a849fb8577794261, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:21,547 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193381546"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193381546"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193381546"}]},"ts":"1682193381546"} 2023-04-22 19:56:21,547 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:56:21,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:21,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e8d85766c3fe2800a849fb8577794261, disabling compactions & flushes 2023-04-22 19:56:21,700 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:21,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:21,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. after waiting 0 ms 2023-04-22 19:56:21,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:21,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-22 19:56:21,704 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:21,704 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:21,704 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e8d85766c3fe2800a849fb8577794261 to self. 2023-04-22 19:56:21,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:21,705 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e8d85766c3fe2800a849fb8577794261, regionState=CLOSED 2023-04-22 19:56:21,706 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193381705"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193381705"}]},"ts":"1682193381705"} 2023-04-22 19:56:21,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-22 19:56:21,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 in 159 msec 2023-04-22 19:56:21,708 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8d85766c3fe2800a849fb8577794261, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,44791,1682193215766; forceNewPlan=false, retain=true 2023-04-22 19:56:21,859 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e8d85766c3fe2800a849fb8577794261, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:21,859 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193381858"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193381858"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193381858"}]},"ts":"1682193381858"} 2023-04-22 19:56:21,860 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:56:22,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e8d85766c3fe2800a849fb8577794261, NAME => 'TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.', STARTKEY => '', ENDKEY => ''} 2023-04-22 19:56:22,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:22,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 19:56:22,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:22,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:22,017 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:22,017 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A 2023-04-22 19:56:22,018 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A 2023-04-22 19:56:22,018 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:56:22,018 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8d85766c3fe2800a849fb8577794261 columnFamilyName A 2023-04-22 19:56:22,019 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(310): Store=e8d85766c3fe2800a849fb8577794261/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:56:22,020 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:22,020 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B 2023-04-22 19:56:22,020 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B 2023-04-22 19:56:22,020 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:56:22,021 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8d85766c3fe2800a849fb8577794261 columnFamilyName B 2023-04-22 19:56:22,021 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(310): Store=e8d85766c3fe2800a849fb8577794261/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:56:22,021 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:22,022 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C 2023-04-22 19:56:22,022 DEBUG [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C 2023-04-22 19:56:22,022 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 19:56:22,022 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8d85766c3fe2800a849fb8577794261 columnFamilyName C 2023-04-22 19:56:22,022 INFO [StoreOpener-e8d85766c3fe2800a849fb8577794261-1] regionserver.HStore(310): Store=e8d85766c3fe2800a849fb8577794261/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 19:56:22,023 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:22,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:22,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 19:56:22,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:22,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e8d85766c3fe2800a849fb8577794261; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66850543, jitterRate=-0.0038492828607559204}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 19:56:22,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:22,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., pid=61, masterSystemTime=1682193382011 2023-04-22 19:56:22,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,031 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e8d85766c3fe2800a849fb8577794261, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,031 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193382031"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682193382031"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682193382031"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682193382031"}]},"ts":"1682193382031"} 2023-04-22 19:56:22,033 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-22 19:56:22,033 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 in 172 msec 2023-04-22 19:56:22,034 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-22 19:56:22,034 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8d85766c3fe2800a849fb8577794261, REOPEN/MOVE in 488 msec 2023-04-22 19:56:22,036 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-22 19:56:22,036 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 492 msec 2023-04-22 19:56:22,037 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 907 msec 2023-04-22 19:56:22,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-22 19:56:22,039 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x6e8824d7 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:56:22,045 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50529bf3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:56:22,045 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x68067de1 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:56:22,050 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@470742aa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:56:22,051 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x45878790 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:56:22,055 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e3503b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:56:22,055 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x1644730f to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:56:22,063 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f06dd50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:56:22,063 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:56:22,067 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:56:22,068 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:56:22,075 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:56:22,075 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x08b57aae to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:56:22,079 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@544a4215, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:56:22,080 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x06799455 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:56:22,087 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25baee3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:56:22,087 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:56:22,091 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:56:22,092 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:53381 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 19:56:22,097 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 19:56:22,106 DEBUG [hconnection-0x26441d56-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:56:22,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:22,107 DEBUG [hconnection-0x175186f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:56:22,107 DEBUG [hconnection-0x5053f2cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:56:22,108 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50646, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:56:22,112 DEBUG [hconnection-0x81af437-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:56:22,114 DEBUG [hconnection-0x560a549c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:56:22,114 DEBUG [hconnection-0x293f6423-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:56:22,115 DEBUG [hconnection-0x5a9b0863-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:56:22,115 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:56:22,116 DEBUG [hconnection-0x1b5d7dc6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:56:22,122 DEBUG [hconnection-0x1a9e1993-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:56:22,122 DEBUG [hconnection-0x4a68f21d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 19:56:22,123 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50690, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:56:22,128 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50640, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:56:22,130 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50628, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:56:22,130 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:56:22,130 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50652, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:56:22,131 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50684, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:56:22,131 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50668, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:56:22,131 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50704, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 19:56:22,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:22,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:22,141 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:22,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:22,141 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:22,142 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,142 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,144 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:22,144 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:22,147 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 19:56:22,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:22,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,147 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:22,147 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,147 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:22,149 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,149 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:22,149 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:22,149 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:22,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:22,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:22,149 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:22,149 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:22,151 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,151 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:22,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220b2306[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:22,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:22,153 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,156 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,156 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,156 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:22,161 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,161 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:22,161 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,161 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,166 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304220ffc5ec523264eaa949803247edaf8e1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220ffc5ec523264eaa949803247edaf8e1 2023-04-22 19:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193442163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:22,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:22,170 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:22,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:22,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:22,170 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:22,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193442165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,170 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:22,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193442166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,173 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193442170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193442169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:22,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:22,177 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,178 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,178 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:22,178 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:22,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193442270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193442271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193442274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193442274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193442274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,278 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:22,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:22,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:22,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:22,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:22,282 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:22,282 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,282 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:22,282 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:22,282 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,282 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,284 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,284 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,284 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,284 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:22,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:22,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:22,286 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:22,286 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,286 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:22,286 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:22,286 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:22,287 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,287 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:22,287 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:22,287 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:22,289 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,289 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,290 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:22,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ff0239e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:22,291 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:22,291 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,292 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:22,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:22,292 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:22,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:22,293 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,293 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:22,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:22,296 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:22,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:22,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:22,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:22,296 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:22,298 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,298 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,298 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,300 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,303 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:22,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:22,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:22,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,304 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:22,304 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:22,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,404 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:22,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:22,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:22,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:22,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:22,407 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:22,407 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,407 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:22,407 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:22,408 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,408 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,409 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,409 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,410 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:22,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:22,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:22,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:22,411 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:22,411 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,411 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:22,411 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:22,411 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:22,412 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,412 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:22,412 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:22,412 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:22,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:22,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:22,412 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:22,414 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,414 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,414 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,414 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e944534[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:22,417 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:22,417 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,418 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:22,418 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,418 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:22,420 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:22,423 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:22,423 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:22,423 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:22,423 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:22,424 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:22,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,428 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,430 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,431 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,434 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,434 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,434 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,434 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,434 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,438 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,438 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,438 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,438 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,438 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:22,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:22,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,438 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:22,439 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:22,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:22,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193442472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193442474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193442475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193442476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193442477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,539 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:22,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:22,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:22,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:22,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:22,542 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:22,542 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,543 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:22,543 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:22,543 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,543 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,550 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,550 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,550 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,550 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:22,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:22,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:22,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:22,552 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,552 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:22,552 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:22,552 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:22,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:22,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:22,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:22,554 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:22,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bd627bf[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:22,556 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:22,556 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,557 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:22,558 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,558 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:22,559 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:22,562 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:22,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:22,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:22,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:22,562 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:22,564 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,564 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,564 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,565 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,569 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:22,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:22,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:22,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,569 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:22,569 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:22,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,575 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=19, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/71a9ee20a6764393b15f4885cd349c1c 2023-04-22 19:56:22,669 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:22,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:22,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:22,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:22,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:22,673 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:22,673 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,673 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:22,673 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:22,674 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,674 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,675 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,675 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,676 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,676 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:22,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:22,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:22,677 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:22,677 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,677 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:22,678 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,678 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:22,678 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:22,678 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:22,679 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,679 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,680 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,680 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:22,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40be30d0[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:22,681 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:22,682 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,683 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:22,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:22,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:22,684 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,684 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:22,685 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:22,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:22,688 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:22,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:22,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:22,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:22,688 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:22,690 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,690 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,690 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,692 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,694 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:22,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:22,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:22,700 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:22,700 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:22,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193442776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193442777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193442778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193442779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:22,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193442781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,800 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:22,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:22,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:22,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:22,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:22,805 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:22,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:22,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:22,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,808 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,808 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,808 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,808 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:22,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:22,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:22,811 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,812 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:22,812 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:22,812 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,812 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:22,812 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:22,812 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,812 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:22,812 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:22,813 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,814 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:22,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e2c178a[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:22,816 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:22,816 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,818 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,818 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:22,818 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:22,818 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:22,818 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,818 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:22,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:22,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:22,819 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:22,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:22,822 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:22,822 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:22,822 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:22,822 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:22,822 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:22,826 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,826 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,826 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,827 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,828 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,833 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:22,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:22,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,833 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:22,833 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:22,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,933 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:22,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:22,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:22,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:22,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:22,937 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:22,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:22,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:22,938 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,938 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,939 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:22,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:22,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:22,941 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,941 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:22,941 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:22,941 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:22,944 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:22,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@228d589e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:22,946 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:22,946 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,947 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,947 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,947 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:22,947 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:22,948 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:22,948 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:22,949 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:22,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:22,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:22,953 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:22,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:22,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:22,954 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:22,954 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:22,955 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,955 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,955 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:22,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,956 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,957 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:22,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:22,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:22,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:22,961 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:22,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:22,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:22,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:22,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:22,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:22,962 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:22,962 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:22,962 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:22,962 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:22,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:22,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/bc1eeebd41584c18aa13d6862defa13a 2023-04-22 19:56:23,000 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/dfc581b1160b484ca38c2f041eb65387 2023-04-22 19:56:23,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/71a9ee20a6764393b15f4885cd349c1c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/71a9ee20a6764393b15f4885cd349c1c 2023-04-22 19:56:23,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/71a9ee20a6764393b15f4885cd349c1c, entries=150, sequenceid=19, filesize=25.1 K 2023-04-22 19:56:23,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/bc1eeebd41584c18aa13d6862defa13a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/bc1eeebd41584c18aa13d6862defa13a 2023-04-22 19:56:23,021 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/bc1eeebd41584c18aa13d6862defa13a, entries=150, sequenceid=19, filesize=11.6 K 2023-04-22 19:56:23,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/dfc581b1160b484ca38c2f041eb65387 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/dfc581b1160b484ca38c2f041eb65387 2023-04-22 19:56:23,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/dfc581b1160b484ca38c2f041eb65387, entries=150, sequenceid=19, filesize=11.6 K 2023-04-22 19:56:23,032 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=107.34 KB/109920 for e8d85766c3fe2800a849fb8577794261 in 885ms, sequenceid=19, compaction requested=false 2023-04-22 19:56:23,032 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-22 19:56:23,033 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:23,062 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:23,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:23,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:23,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:23,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:23,065 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:23,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:23,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:23,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,067 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,067 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:23,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:23,068 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,068 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:23,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:23,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:23,069 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,069 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:23,069 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 19:56:23,070 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:23,070 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:23,070 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:23,070 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:23,070 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:23,070 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:23,097 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422b9af0cc985a942718de1a892aca8a977 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b9af0cc985a942718de1a892aca8a977 2023-04-22 19:56:23,123 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=38, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/46a8ab2dc6084416967460f17aab9ded 2023-04-22 19:56:23,138 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/6fc8ef7385fe4782a8cf9ea2f46eec1d 2023-04-22 19:56:23,158 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/b2c46874a9004108bd16d8ebeb1caa23 2023-04-22 19:56:23,162 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/46a8ab2dc6084416967460f17aab9ded as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/46a8ab2dc6084416967460f17aab9ded 2023-04-22 19:56:23,167 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/46a8ab2dc6084416967460f17aab9ded, entries=150, sequenceid=38, filesize=25.1 K 2023-04-22 19:56:23,168 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/6fc8ef7385fe4782a8cf9ea2f46eec1d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/6fc8ef7385fe4782a8cf9ea2f46eec1d 2023-04-22 19:56:23,172 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/6fc8ef7385fe4782a8cf9ea2f46eec1d, entries=150, sequenceid=38, filesize=11.6 K 2023-04-22 19:56:23,173 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/b2c46874a9004108bd16d8ebeb1caa23 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b2c46874a9004108bd16d8ebeb1caa23 2023-04-22 19:56:23,180 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b2c46874a9004108bd16d8ebeb1caa23, entries=150, sequenceid=38, filesize=11.6 K 2023-04-22 19:56:23,183 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for e8d85766c3fe2800a849fb8577794261 in 114ms, sequenceid=38, compaction requested=false 2023-04-22 19:56:23,184 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:23,184 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,184 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:23,184 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:23,184 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,185 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:23,185 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,186 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:23,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@286c78af[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:23,188 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:23,188 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,189 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,190 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:23,190 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:23,190 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:23,190 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,190 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:23,192 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:23,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:23,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:23,195 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:23,195 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:23,195 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:23,196 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:23,196 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:23,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:23,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,198 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:23,202 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:23,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:23,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:23,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:23,202 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:23,203 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:23,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:23,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:23,303 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:23,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:23,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:23,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:23,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:23,308 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:23,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:23,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:23,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,309 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:23,310 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:23,310 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,310 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:23,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:23,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:23,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:23,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:23,314 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:56:23,316 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:23,316 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:23,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:23,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:23,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:23,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:23,324 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,324 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:23,324 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:23,324 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:23,327 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,327 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:23,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1965e5c2[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:23,330 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:23,330 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,331 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:23,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:23,331 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:23,332 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,332 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:23,333 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:23,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:23,337 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:23,337 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:23,337 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:23,337 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:23,337 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:23,339 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,339 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:23,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,339 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,340 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193383319, max timestamp=1682193383339Num uniques 859; 2023-04-22 19:56:23,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193383319, max timestamp=1682193383339Num uniques 859; 2023-04-22 19:56:23,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:56:23,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193383319, max timestamp=1682193383339Num uniques 859; 2023-04-22 19:56:23,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:56:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,344 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:23,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:23,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:23,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:23,349 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:23,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:23,349 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:23,349 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,349 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:23,349 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:23,349 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,349 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,349 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,349 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193443369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193443372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193443373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,378 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422d9d3b0b9d355447ba5e59fe11105cbe6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d9d3b0b9d355447ba5e59fe11105cbe6 2023-04-22 19:56:23,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193443373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193443373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,394 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/fbdf1ff6f00541178ec185d35e6a12f5 2023-04-22 19:56:23,449 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:23,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:23,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:23,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:23,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:23,454 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:23,454 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,454 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:23,454 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:23,456 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,456 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,460 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:23,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:23,461 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,461 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:23,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:23,462 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:23,462 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,462 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:23,462 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:23,462 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:23,462 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,462 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:23,462 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:23,462 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,462 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:23,464 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,464 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,464 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,464 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:23,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@472b2123[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:23,467 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:23,467 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,469 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,469 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:23,469 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:23,469 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:23,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,469 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:23,470 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:23,470 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:23,470 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:23,470 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:23,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:23,474 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:23,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:23,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:23,474 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:23,474 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:23,476 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,476 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193443474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,476 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:23,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,479 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193443478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193443479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193443479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193443480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:23,484 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:23,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:23,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:23,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,487 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,488 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,488 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:23,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:23,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:23,488 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:23,488 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:23,588 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:23,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:23,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:23,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:23,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:23,592 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:23,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:23,592 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:23,600 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,601 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,603 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:23,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:23,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,604 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:23,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:23,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:23,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:23,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:23,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:23,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:23,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:23,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:23,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:23,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:23,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:23,607 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:23,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:23,609 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,609 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:23,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@faac8f0[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:23,611 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:23,611 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,612 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,613 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:23,613 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:23,613 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:23,613 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,613 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:23,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:23,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:23,617 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:23,617 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:23,617 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:23,617 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:23,617 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:23,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,620 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,622 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,624 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,624 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,624 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,624 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,624 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,624 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,624 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,624 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:23,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:23,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:23,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:23,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:23,624 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:23,624 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:23,625 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:23,625 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:23,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:23,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:23,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193443678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193443682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193443683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193443684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193443684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,725 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:23,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:23,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:23,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:23,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:23,729 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:23,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:23,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:23,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,729 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,730 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:23,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:23,731 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,731 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,731 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:23,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:23,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:23,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:23,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:23,732 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:23,733 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:23,733 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,733 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:23,733 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:23,733 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,733 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:23,733 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:23,733 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,733 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:23,733 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:23,735 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,735 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,736 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,736 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:23,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:23,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44c870aa[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:23,737 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:23,737 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,741 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,741 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:23,741 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:23,741 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:23,741 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,741 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:23,742 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:23,742 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,742 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:23,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,742 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:23,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:23,745 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:23,745 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:23,745 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:23,745 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:23,745 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:23,747 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,747 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:23,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,747 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:23,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:23,749 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,750 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:23,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:23,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,753 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:23,753 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:23,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:23,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:23,753 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:23,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:23,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,754 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:23,754 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:23,754 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:23,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:23,754 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,784 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 19:56:23,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/0109ddb8f20e4b7bac204baad88662c4 2023-04-22 19:56:23,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/c0b1314510304a01b4a11f388a4eca1e 2023-04-22 19:56:23,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/fbdf1ff6f00541178ec185d35e6a12f5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/fbdf1ff6f00541178ec185d35e6a12f5 2023-04-22 19:56:23,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/fbdf1ff6f00541178ec185d35e6a12f5, entries=150, sequenceid=49, filesize=25.1 K 2023-04-22 19:56:23,829 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/0109ddb8f20e4b7bac204baad88662c4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0109ddb8f20e4b7bac204baad88662c4 2023-04-22 19:56:23,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0109ddb8f20e4b7bac204baad88662c4, entries=150, sequenceid=49, filesize=11.6 K 2023-04-22 19:56:23,833 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/c0b1314510304a01b4a11f388a4eca1e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/c0b1314510304a01b4a11f388a4eca1e 2023-04-22 19:56:23,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/c0b1314510304a01b4a11f388a4eca1e, entries=150, sequenceid=49, filesize=11.6 K 2023-04-22 19:56:23,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for e8d85766c3fe2800a849fb8577794261 in 523ms, sequenceid=49, compaction requested=true 2023-04-22 19:56:23,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:23,837 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:23,837 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:23,837 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:56:23,837 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:56:23,837 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:23,838 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:56:23,838 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:56:23,838 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/A is initiating minor compaction (all files) 2023-04-22 19:56:23,838 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/B is initiating minor compaction (all files) 2023-04-22 19:56:23,838 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/A in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,838 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/B in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,838 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/71a9ee20a6764393b15f4885cd349c1c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/46a8ab2dc6084416967460f17aab9ded, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/fbdf1ff6f00541178ec185d35e6a12f5] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=75.4 K 2023-04-22 19:56:23,838 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/bc1eeebd41584c18aa13d6862defa13a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/6fc8ef7385fe4782a8cf9ea2f46eec1d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0109ddb8f20e4b7bac204baad88662c4] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=34.8 K 2023-04-22 19:56:23,838 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 71a9ee20a6764393b15f4885cd349c1c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682193382143 2023-04-22 19:56:23,839 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting bc1eeebd41584c18aa13d6862defa13a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682193382143 2023-04-22 19:56:23,839 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 46a8ab2dc6084416967460f17aab9ded, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1682193382159 2023-04-22 19:56:23,839 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting fbdf1ff6f00541178ec185d35e6a12f5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682193383305 2023-04-22 19:56:23,839 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 6fc8ef7385fe4782a8cf9ea2f46eec1d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1682193382159 2023-04-22 19:56:23,840 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0109ddb8f20e4b7bac204baad88662c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682193383305 2023-04-22 19:56:23,847 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#B#compaction#489 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:23,847 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#A#compaction#488 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:23,854 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:23,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:23,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:23,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:23,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:23,861 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:23,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:23,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:23,861 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,864 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,864 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:23,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:23,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:23,864 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,864 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:23,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:23,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:23,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:23,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:23,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:23,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:23,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:23,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:23,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:23,868 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:23,868 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:23,868 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:23,868 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:23,868 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-22 19:56:23,869 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:23,869 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:23,869 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:23,869 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:23,869 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:23,869 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:23,902 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304220404a5968688453b9eff4223a27f344f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220404a5968688453b9eff4223a27f344f 2023-04-22 19:56:23,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:23,987 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:23,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193443990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193443991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193443991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193443996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:23,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:23,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193443996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193444096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193444098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193444098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193444098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193444099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,269 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/039044d3bdab43fbad17f0c43d5ef85b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/039044d3bdab43fbad17f0c43d5ef85b 2023-04-22 19:56:24,272 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/6fe8d27774984c058e4478fa20ce8f22 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/6fe8d27774984c058e4478fa20ce8f22 2023-04-22 19:56:24,277 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8d85766c3fe2800a849fb8577794261/A of e8d85766c3fe2800a849fb8577794261 into 6fe8d27774984c058e4478fa20ce8f22(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:24,277 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:24,277 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/A, priority=13, startTime=1682193383837; duration=0sec 2023-04-22 19:56:24,277 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:24,277 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:56:24,278 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8d85766c3fe2800a849fb8577794261/B of e8d85766c3fe2800a849fb8577794261 into 039044d3bdab43fbad17f0c43d5ef85b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:24,278 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:24,278 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/B, priority=13, startTime=1682193383837; duration=0sec 2023-04-22 19:56:24,278 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:24,280 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:56:24,280 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/C is initiating minor compaction (all files) 2023-04-22 19:56:24,280 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/C in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:24,280 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/dfc581b1160b484ca38c2f041eb65387, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b2c46874a9004108bd16d8ebeb1caa23, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/c0b1314510304a01b4a11f388a4eca1e] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=34.8 K 2023-04-22 19:56:24,281 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting dfc581b1160b484ca38c2f041eb65387, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682193382143 2023-04-22 19:56:24,281 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting b2c46874a9004108bd16d8ebeb1caa23, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1682193382159 2023-04-22 19:56:24,281 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c0b1314510304a01b4a11f388a4eca1e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682193383305 2023-04-22 19:56:24,288 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#C#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:24,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193444301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193444301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193444302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193444302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193444303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,326 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=86, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/438b6a3f24ef47debf744856e6590c84 2023-04-22 19:56:24,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193444604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193444606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193444606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193444606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:24,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193444606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:24,697 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/59d71275b4974794a82ba6cafa9e590c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/59d71275b4974794a82ba6cafa9e590c 2023-04-22 19:56:24,700 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8d85766c3fe2800a849fb8577794261/C of e8d85766c3fe2800a849fb8577794261 into 59d71275b4974794a82ba6cafa9e590c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:24,701 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:24,701 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/C, priority=13, startTime=1682193383837; duration=0sec 2023-04-22 19:56:24,701 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:24,742 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/0f1b26369f7d4fcc8eb220e1f62cc7b1 2023-04-22 19:56:25,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:25,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:25,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193445108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193445109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:25,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193445109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:25,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193445111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:25,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193445112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,154 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/928d11612f0c44f7ad7396df5688e32c 2023-04-22 19:56:25,157 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/438b6a3f24ef47debf744856e6590c84 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/438b6a3f24ef47debf744856e6590c84 2023-04-22 19:56:25,161 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/438b6a3f24ef47debf744856e6590c84, entries=150, sequenceid=86, filesize=25.1 K 2023-04-22 19:56:25,162 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/0f1b26369f7d4fcc8eb220e1f62cc7b1 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0f1b26369f7d4fcc8eb220e1f62cc7b1 2023-04-22 19:56:25,169 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0f1b26369f7d4fcc8eb220e1f62cc7b1, entries=150, sequenceid=86, filesize=11.6 K 2023-04-22 19:56:25,170 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/928d11612f0c44f7ad7396df5688e32c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/928d11612f0c44f7ad7396df5688e32c 2023-04-22 19:56:25,173 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/928d11612f0c44f7ad7396df5688e32c, entries=150, sequenceid=86, filesize=11.6 K 2023-04-22 19:56:25,174 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for e8d85766c3fe2800a849fb8577794261 in 1306ms, sequenceid=86, compaction requested=false 2023-04-22 19:56:25,174 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:25,174 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:25,174 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:25,174 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:25,174 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,174 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:25,174 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:25,179 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,179 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:25,179 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,179 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:25,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:25,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7235c604[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:25,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:25,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,182 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,182 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,182 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:25,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:25,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:25,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:25,184 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:25,184 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:25,184 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:25,184 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:25,187 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:25,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:25,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:25,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:25,187 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:25,189 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,189 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,189 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:25,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:25,190 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,192 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:25,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,194 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:25,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:25,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:25,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:25,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:25,194 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:25,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,194 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:25,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:25,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:25,294 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:25,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:25,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:25,298 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:25,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:25,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:25,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:25,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,298 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,300 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:25,300 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:25,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:25,300 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,300 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:25,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:25,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:25,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:25,301 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:25,301 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:25,301 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:25,301 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:25,301 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:25,301 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:25,302 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:25,302 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 19:56:25,302 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:25,302 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:25,302 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:25,302 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:25,302 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:25,302 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:25,325 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422f01e9365d78a406092a460a3024cc54c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f01e9365d78a406092a460a3024cc54c 2023-04-22 19:56:25,331 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/145038b32a9d488fb87cf0a27cf63393 2023-04-22 19:56:25,339 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/92d0a54043b94838b75ce172f58c3d99 2023-04-22 19:56:25,750 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/b140cffa802a405e95f61f8e20595f2e 2023-04-22 19:56:25,754 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/145038b32a9d488fb87cf0a27cf63393 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/145038b32a9d488fb87cf0a27cf63393 2023-04-22 19:56:25,759 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/145038b32a9d488fb87cf0a27cf63393, entries=100, sequenceid=98, filesize=18.4 K 2023-04-22 19:56:25,759 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/92d0a54043b94838b75ce172f58c3d99 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/92d0a54043b94838b75ce172f58c3d99 2023-04-22 19:56:25,763 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/92d0a54043b94838b75ce172f58c3d99, entries=100, sequenceid=98, filesize=9.3 K 2023-04-22 19:56:25,763 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/b140cffa802a405e95f61f8e20595f2e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b140cffa802a405e95f61f8e20595f2e 2023-04-22 19:56:25,768 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b140cffa802a405e95f61f8e20595f2e, entries=100, sequenceid=98, filesize=9.3 K 2023-04-22 19:56:25,773 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for e8d85766c3fe2800a849fb8577794261 in 471ms, sequenceid=98, compaction requested=true 2023-04-22 19:56:25,773 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:25,773 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:25,774 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:25,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,776 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:25,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,776 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:25,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:25,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c09c358[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:25,778 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:25,778 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,779 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:25,779 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,779 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:25,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:25,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:25,780 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,780 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:25,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:25,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:25,783 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:25,783 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:25,783 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:25,783 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:25,783 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:25,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:25,785 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:25,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:25,785 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,787 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,795 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:25,795 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,795 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:25,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:25,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:25,795 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:25,795 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:25,795 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:25,795 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:25,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:25,796 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,796 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:25,796 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,796 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,796 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,896 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:25,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:25,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:25,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:25,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:25,900 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:25,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:25,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:25,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:25,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,900 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,901 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:25,901 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:25,902 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,902 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:25,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:25,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:25,903 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:25,903 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:25,903 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:25,903 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:25,903 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:25,903 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:25,903 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:25,904 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:25,904 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:25,905 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,905 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:25,905 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,905 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:25,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:25,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@353d7b61[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:25,907 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:25,907 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,908 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,908 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:25,908 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:25,908 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:25,908 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,909 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:25,911 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,911 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:25,911 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:25,911 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:25,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:25,914 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:25,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:25,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:25,914 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:25,914 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:25,915 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,916 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,916 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:25,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:25,916 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:25,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,917 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,920 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:25,920 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:25,920 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:25,920 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,920 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:25,920 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:25,920 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,920 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:25,920 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:25,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:25,920 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:25,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:25,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:25,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:25,921 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:25,921 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:26,027 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:26,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:26,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:26,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:26,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:26,033 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:26,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:26,033 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:26,034 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,034 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,035 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,035 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,036 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,036 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:26,037 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:26,037 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,037 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:26,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:26,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:26,044 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,044 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:26,044 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:26,044 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:26,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:26,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:26,044 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:26,046 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,046 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:26,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3277104c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:26,049 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:26,049 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,050 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,050 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:26,050 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:26,050 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:26,050 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,050 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:26,057 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:26,057 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,057 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,058 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:26,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:26,060 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:26,060 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:26,060 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:26,060 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:26,060 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:26,064 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,064 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,064 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,068 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:26,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,069 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:26,069 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:26,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:26,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:26,139 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:56:26,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:26,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:26,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:26,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:26,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:26,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:26,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193446159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193386154, max timestamp=1682193386162Num uniques 859; 2023-04-22 19:56:26,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:56:26,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193446161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193386158, max timestamp=1682193386162Num uniques 859; 2023-04-22 19:56:26,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:26,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193386158, max timestamp=1682193386162Num uniques 859; 2023-04-22 19:56:26,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:26,169 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:26,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:26,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:26,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:26,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:26,172 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:26,172 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,172 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:26,172 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:26,173 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,173 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,174 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,174 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,174 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,174 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:26,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:26,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:26,175 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:26,175 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,175 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:26,175 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:26,175 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:26,175 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,176 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:26,176 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:26,176 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:26,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:26,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:26,176 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:26,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,177 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:26,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e2ef5e0[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:26,179 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:26,179 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,181 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:26,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:26,182 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:26,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:26,185 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:26,185 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:26,185 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:26,185 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:26,185 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:26,186 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,186 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,186 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,189 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,191 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,191 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,191 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,191 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,191 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,191 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:26,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:26,191 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,192 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,192 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,191 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,192 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:26,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193446188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,192 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:26,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193446189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193446189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193446265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193446266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,292 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:26,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:26,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:26,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193446293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:26,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:26,296 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:26,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:26,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:26,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,296 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193446294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193446294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,303 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,303 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,304 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:26,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:26,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:26,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:26,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:26,306 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,306 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:26,306 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:26,306 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:26,307 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,307 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,308 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:26,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c9fe87e[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:26,309 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:26,309 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,310 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:26,311 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,311 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:26,312 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:26,312 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,312 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:26,312 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:26,314 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:26,315 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:26,315 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:26,315 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:26,315 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:26,317 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,318 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,318 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,318 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:26,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:26,323 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,322 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,323 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:26,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:26,323 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:26,423 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:26,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:26,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:26,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:26,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:26,427 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:26,427 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,427 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:26,427 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:26,427 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,427 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,429 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,429 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,429 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,429 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:26,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:26,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:26,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:26,430 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:26,430 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,430 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:26,431 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,431 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:26,431 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:26,431 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:26,433 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:26,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12a764ff[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:26,435 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:26,435 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,438 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:26,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:26,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:26,438 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,439 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,440 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:26,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:26,443 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:26,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:26,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:26,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:26,443 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:26,446 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,446 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,446 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,448 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:26,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:26,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:26,453 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,454 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:26,454 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:26,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,454 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,454 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193446469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193446469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193446497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193446498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193446499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,554 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:26,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:26,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:26,557 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304223ebba83edba54af4a814b3524bf94f3f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223ebba83edba54af4a814b3524bf94f3f 2023-04-22 19:56:26,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:26,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:26,558 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:26,558 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,558 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:26,558 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:26,558 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,558 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,560 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,560 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,560 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,560 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:26,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:26,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:26,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:26,561 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:26,561 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,561 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:26,562 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,562 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:26,562 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:26,562 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:26,563 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,563 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,564 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=111, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/c81e20131e1f48aa972ffa459f67fc8c 2023-04-22 19:56:26,564 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,564 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:26,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d8e397[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:26,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:26,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,567 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,567 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:26,567 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:26,567 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,567 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:26,567 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:26,570 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:26,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:26,573 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:26,573 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:26,573 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:26,573 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:26,573 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:26,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,575 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,576 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,579 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,579 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,579 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:26,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:26,579 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,579 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,579 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,579 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,579 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:26,579 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,579 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,579 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:26,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,579 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,680 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:26,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:26,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:26,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:26,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:26,683 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:26,683 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,683 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:26,683 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:26,684 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,684 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,685 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,685 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,685 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,685 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:26,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:26,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:26,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:26,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:26,686 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:26,687 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,687 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:26,687 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:26,687 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:26,690 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,690 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,690 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,690 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:26,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1408499[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:26,692 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:26,692 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,693 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:26,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:26,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:26,693 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,694 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:26,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:26,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:26,698 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:26,698 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:26,698 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:26,698 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:26,698 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:26,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,700 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,702 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,705 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,705 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,705 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,706 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,705 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:26,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:26,706 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,706 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,706 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,706 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,706 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,706 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:26,706 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193446771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193446773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193446800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193446801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:26,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193446804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,806 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:26,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:26,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:26,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:26,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:26,810 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:26,810 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,810 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:26,810 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:26,810 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,810 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,811 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,811 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,812 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,812 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:26,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:26,813 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:26,813 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,813 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:26,814 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,814 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:26,814 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:26,814 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:26,815 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32af50d7[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:26,818 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:26,818 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,819 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,819 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:26,819 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,822 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:26,825 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:26,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:26,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:26,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:26,825 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:26,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,827 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,829 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,834 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,834 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,834 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,834 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,834 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,834 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,834 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,834 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,834 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:26,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:26,835 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:26,835 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:26,834 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,935 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:26,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:26,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:26,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:26,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:26,938 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:26,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:26,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:26,940 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,940 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,943 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,943 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,944 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,944 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:26,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:26,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:26,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:26,945 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,945 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:26,945 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:26,945 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,947 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,948 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,948 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b065ae1[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:26,949 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:26,949 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,950 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:26,951 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,951 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:26,952 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:26,955 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:26,955 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:26,955 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:26,955 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:26,955 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:26,957 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:26,957 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,958 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,958 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,960 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,962 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:26,962 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:26,963 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:26,963 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,963 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:26,963 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:26,963 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,963 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:26,963 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:26,963 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:26,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:26,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:26,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:26,963 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:26,963 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:26,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/ada350b365784baa93a85ef40ba50f5c 2023-04-22 19:56:26,994 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/ce932dce1df54501920060684c5ebd06 2023-04-22 19:56:26,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/c81e20131e1f48aa972ffa459f67fc8c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c81e20131e1f48aa972ffa459f67fc8c 2023-04-22 19:56:27,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c81e20131e1f48aa972ffa459f67fc8c, entries=400, sequenceid=111, filesize=59.0 K 2023-04-22 19:56:27,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/ada350b365784baa93a85ef40ba50f5c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ada350b365784baa93a85ef40ba50f5c 2023-04-22 19:56:27,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ada350b365784baa93a85ef40ba50f5c, entries=150, sequenceid=111, filesize=11.6 K 2023-04-22 19:56:27,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/ce932dce1df54501920060684c5ebd06 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ce932dce1df54501920060684c5ebd06 2023-04-22 19:56:27,024 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ce932dce1df54501920060684c5ebd06, entries=150, sequenceid=111, filesize=11.6 K 2023-04-22 19:56:27,024 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for e8d85766c3fe2800a849fb8577794261 in 885ms, sequenceid=111, compaction requested=true 2023-04-22 19:56:27,024 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:27,025 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:27,025 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:56:27,025 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:56:27,025 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:27,025 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:27,026 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 130847 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:56:27,026 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/A is initiating minor compaction (all files) 2023-04-22 19:56:27,026 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/A in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:27,026 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/6fe8d27774984c058e4478fa20ce8f22, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/438b6a3f24ef47debf744856e6590c84, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/145038b32a9d488fb87cf0a27cf63393, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c81e20131e1f48aa972ffa459f67fc8c] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=127.8 K 2023-04-22 19:56:27,026 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:56:27,026 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/B is initiating minor compaction (all files) 2023-04-22 19:56:27,026 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/B in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:27,027 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/039044d3bdab43fbad17f0c43d5ef85b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0f1b26369f7d4fcc8eb220e1f62cc7b1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/92d0a54043b94838b75ce172f58c3d99, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ada350b365784baa93a85ef40ba50f5c] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=44.2 K 2023-04-22 19:56:27,027 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 6fe8d27774984c058e4478fa20ce8f22, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682193383305 2023-04-22 19:56:27,027 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 039044d3bdab43fbad17f0c43d5ef85b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682193383305 2023-04-22 19:56:27,028 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 438b6a3f24ef47debf744856e6590c84, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682193383368 2023-04-22 19:56:27,029 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0f1b26369f7d4fcc8eb220e1f62cc7b1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682193383368 2023-04-22 19:56:27,029 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 145038b32a9d488fb87cf0a27cf63393, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682193383991 2023-04-22 19:56:27,029 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 92d0a54043b94838b75ce172f58c3d99, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682193383991 2023-04-22 19:56:27,029 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c81e20131e1f48aa972ffa459f67fc8c, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682193386129 2023-04-22 19:56:27,029 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ada350b365784baa93a85ef40ba50f5c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682193386129 2023-04-22 19:56:27,037 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#B#compaction#501 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:27,057 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#A#compaction#500 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:27,063 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:27,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:27,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:27,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:27,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:27,070 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:27,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:27,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:27,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:27,071 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:27,071 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:27,072 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:27,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:27,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:27,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:27,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:27,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:27,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:27,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:27,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:27,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:27,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:27,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:27,077 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-04-22 19:56:27,077 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:27,077 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:27,077 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:27,077 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:27,077 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:27,077 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:27,082 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/a689097420c54720add9e01c901a0685 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/a689097420c54720add9e01c901a0685 2023-04-22 19:56:27,094 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8d85766c3fe2800a849fb8577794261/B of e8d85766c3fe2800a849fb8577794261 into a689097420c54720add9e01c901a0685(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:27,094 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:27,094 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/B, priority=12, startTime=1682193387025; duration=0sec 2023-04-22 19:56:27,094 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:27,094 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:56:27,095 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:56:27,096 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/C is initiating minor compaction (all files) 2023-04-22 19:56:27,096 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/C in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:27,096 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/59d71275b4974794a82ba6cafa9e590c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/928d11612f0c44f7ad7396df5688e32c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b140cffa802a405e95f61f8e20595f2e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ce932dce1df54501920060684c5ebd06] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=44.2 K 2023-04-22 19:56:27,096 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 59d71275b4974794a82ba6cafa9e590c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682193383305 2023-04-22 19:56:27,097 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 928d11612f0c44f7ad7396df5688e32c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682193383368 2023-04-22 19:56:27,097 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting b140cffa802a405e95f61f8e20595f2e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682193383991 2023-04-22 19:56:27,097 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ce932dce1df54501920060684c5ebd06, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682193386129 2023-04-22 19:56:27,105 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#C#compaction#503 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:27,117 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/7bc4b56bad464616b4a92575f8be33f4 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/7bc4b56bad464616b4a92575f8be33f4 2023-04-22 19:56:27,122 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8d85766c3fe2800a849fb8577794261/C of e8d85766c3fe2800a849fb8577794261 into 7bc4b56bad464616b4a92575f8be33f4(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:27,122 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:27,122 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/C, priority=12, startTime=1682193387025; duration=0sec 2023-04-22 19:56:27,122 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:27,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:27,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:27,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:27,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193447304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:27,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193447306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:27,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193447307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:27,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193447308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:27,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193447308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:27,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193447408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:27,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193447409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,505 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422c9b75bf2bbf14622aee57f268a83ea51 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c9b75bf2bbf14622aee57f268a83ea51 2023-04-22 19:56:27,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:27,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193447612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:27,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193447614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,904 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/a984a31c437740769b39a6e4d6611017 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/a984a31c437740769b39a6e4d6611017 2023-04-22 19:56:27,908 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8d85766c3fe2800a849fb8577794261/A of e8d85766c3fe2800a849fb8577794261 into a984a31c437740769b39a6e4d6611017(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:27,908 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:27,908 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/A, priority=12, startTime=1682193387025; duration=0sec 2023-04-22 19:56:27,909 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:27,911 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/45be71951bde41118df6a1182c18fb66 2023-04-22 19:56:27,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:27,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193447914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,919 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/f1ff711ed6914a829afff5541ed92c13 2023-04-22 19:56:27,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:27,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193447919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,927 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/e6f1e7a5a70246a19df914a682e5ff7a 2023-04-22 19:56:27,930 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/45be71951bde41118df6a1182c18fb66 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/45be71951bde41118df6a1182c18fb66 2023-04-22 19:56:27,933 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/45be71951bde41118df6a1182c18fb66, entries=150, sequenceid=147, filesize=25.3 K 2023-04-22 19:56:27,934 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/f1ff711ed6914a829afff5541ed92c13 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/f1ff711ed6914a829afff5541ed92c13 2023-04-22 19:56:27,937 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/f1ff711ed6914a829afff5541ed92c13, entries=150, sequenceid=147, filesize=11.8 K 2023-04-22 19:56:27,938 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/e6f1e7a5a70246a19df914a682e5ff7a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e6f1e7a5a70246a19df914a682e5ff7a 2023-04-22 19:56:27,941 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e6f1e7a5a70246a19df914a682e5ff7a, entries=150, sequenceid=147, filesize=11.8 K 2023-04-22 19:56:27,941 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=53.67 KB/54960 for e8d85766c3fe2800a849fb8577794261 in 865ms, sequenceid=147, compaction requested=false 2023-04-22 19:56:27,942 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:27,942 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:27,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:27,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:27,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:27,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:27,943 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,943 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:27,943 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:27,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:27,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:27,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449dd64d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:27,945 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:27,945 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:27,947 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:27,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:27,947 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:27,947 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:27,947 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:27,947 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,947 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:27,949 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:27,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:27,953 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:27,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:27,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:27,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:27,953 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:27,959 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:27,959 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:27,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:27,959 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:27,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:27,960 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:27,962 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:27,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:27,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:27,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:27,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:27,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:27,966 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:27,966 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:27,966 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:27,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:27,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:27,967 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:27,967 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:27,967 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:27,967 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:27,967 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:28,067 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:28,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:28,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:28,070 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:28,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:28,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:28,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:28,071 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:28,071 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:28,073 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:28,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:28,074 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,074 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:28,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:28,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:28,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:28,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:28,075 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:28,075 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:28,075 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 19:56:28,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:28,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:28,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:28,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:28,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:28,076 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:28,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:28,316 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:28,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193448334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193388327, max timestamp=1682193388334Num uniques 859; 2023-04-22 19:56:28,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193388327, max timestamp=1682193388334Num uniques 859; 2023-04-22 19:56:28,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682193388327, max timestamp=1682193388334Num uniques 859; 2023-04-22 19:56:28,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:56:28,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:56:28,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 19:56:28,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193448361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193448362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193448419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193448424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193448437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193448465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193448466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,493 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422a40238fcc11347298b41a9aa30c64a55 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a40238fcc11347298b41a9aa30c64a55 2023-04-22 19:56:28,500 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=161, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/63dc634ff8794a288cb4f157bc667745 2023-04-22 19:56:28,509 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/3bb364ccb15f41de96c002b4cfbf3948 2023-04-22 19:56:28,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193448641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193448669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193448672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,918 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/070dfbbd4afb4a21815004a6de27f1a7 2023-04-22 19:56:28,925 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/63dc634ff8794a288cb4f157bc667745 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/63dc634ff8794a288cb4f157bc667745 2023-04-22 19:56:28,929 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/63dc634ff8794a288cb4f157bc667745, entries=150, sequenceid=161, filesize=25.3 K 2023-04-22 19:56:28,929 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/3bb364ccb15f41de96c002b4cfbf3948 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/3bb364ccb15f41de96c002b4cfbf3948 2023-04-22 19:56:28,933 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/3bb364ccb15f41de96c002b4cfbf3948, entries=150, sequenceid=161, filesize=11.8 K 2023-04-22 19:56:28,933 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/070dfbbd4afb4a21815004a6de27f1a7 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/070dfbbd4afb4a21815004a6de27f1a7 2023-04-22 19:56:28,937 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/070dfbbd4afb4a21815004a6de27f1a7, entries=150, sequenceid=161, filesize=11.8 K 2023-04-22 19:56:28,937 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e8d85766c3fe2800a849fb8577794261 in 862ms, sequenceid=161, compaction requested=true 2023-04-22 19:56:28,938 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:28,938 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:28,939 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:28,940 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:28,940 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:28,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:28,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@417ce60b[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:28,941 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:28,941 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:28,942 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:28,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:28,943 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,943 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:28,944 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:28,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:28,947 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:28,947 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:28,947 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:28,947 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:28,948 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:28,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:28,948 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-22 19:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:28,952 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:28,952 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:28,952 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:28,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:28,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:28,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,954 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:28,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:28,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:28,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:28,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:28,960 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:28,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:28,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:28,960 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:28,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:28,961 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:28,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:28,961 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:28,961 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:28,964 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422cd00dd7e50d2495586c20cc8585ec2b4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cd00dd7e50d2495586c20cc8585ec2b4 2023-04-22 19:56:28,971 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=196, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/088baf6b8a1c451691b3b0f9c7af30ec 2023-04-22 19:56:28,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193448974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193448974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:28,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:28,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193448975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,061 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:29,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:29,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:29,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:29,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:29,064 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:29,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:29,064 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:29,065 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,065 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,066 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,066 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:29,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:29,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:29,067 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,067 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:29,068 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:29,068 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:29,070 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@99d2f1b[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:29,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:29,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,073 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:29,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:29,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,073 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:29,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:29,074 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,074 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,074 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:29,077 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:29,077 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:29,077 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:29,077 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:29,077 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:29,078 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,078 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,078 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,079 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,080 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193449078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,083 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,083 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,083 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,083 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,083 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:29,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:29,083 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:29,084 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,083 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,083 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,084 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,084 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:29,084 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:29,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,184 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:29,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:29,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:29,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:29,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:29,187 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:29,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:29,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:29,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,189 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,189 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,189 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,189 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:29,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:29,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:29,191 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,191 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:29,191 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:29,191 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:29,192 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,193 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:29,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20e5bd9b[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:29,194 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:29,194 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,195 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:29,196 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,196 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:29,198 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,198 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:29,198 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,198 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:29,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:29,200 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:29,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:29,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:29,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:29,200 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:29,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,205 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,206 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,208 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:29,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:29,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,208 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:29,209 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:29,208 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,209 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,209 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,209 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:29,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193449283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,309 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:29,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:29,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:29,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:29,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:29,312 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:29,312 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,312 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:29,312 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:29,313 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,313 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,314 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,314 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,314 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,314 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:29,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:29,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:29,316 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,316 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:29,316 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:29,316 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:29,318 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,319 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,319 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:29,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae02874[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:29,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:29,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,321 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,321 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:29,321 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:29,321 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:29,321 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,321 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:29,322 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:29,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:29,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:29,325 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:29,325 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:29,325 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:29,325 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:29,325 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:29,326 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,326 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,326 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,328 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,329 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,331 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,331 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,332 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,332 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:29,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:29,332 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,332 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,332 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,332 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,332 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:29,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,332 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,332 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:29,332 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,382 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/0c420fb0d2e848c487a4b06342865ee5 2023-04-22 19:56:29,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:29,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193449427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:29,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193449429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,432 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:29,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:29,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:29,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:29,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:29,436 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:29,436 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,436 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:29,436 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:29,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,438 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,438 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,438 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,438 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:29,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:29,439 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:29,439 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:29,440 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,440 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:29,440 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:29,440 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:29,444 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,444 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c0adb9d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:29,446 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:29,446 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,449 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:29,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:29,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:29,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,449 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:29,450 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,451 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:29,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:29,453 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:29,454 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:29,454 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:29,454 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:29,454 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:29,455 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,455 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,455 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,456 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,457 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,459 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,459 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,459 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,462 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:29,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:29,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,462 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:29,462 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:29,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:29,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193449478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:29,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193449480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,562 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:29,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:29,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:29,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:29,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:29,566 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:29,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:29,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:29,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,568 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,568 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,569 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,569 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:29,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:29,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:29,570 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,570 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:29,570 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:29,570 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:29,571 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:29,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c8fdcf0[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:29,573 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:29,573 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:29,574 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,574 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:29,575 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:29,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:29,578 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:29,578 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:29,578 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:29,578 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:29,578 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:29,580 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,580 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,580 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,581 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,582 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,584 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:29,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:29,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,584 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:29,584 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:29,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:29,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:29,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193449585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,684 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:29,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:29,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:29,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:29,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:29,688 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:29,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:29,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:29,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,689 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,690 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,690 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:29,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:29,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:29,691 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:29,691 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,691 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:29,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:29,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:29,692 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:29,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@418ba99f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:29,696 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:29,696 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,698 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,698 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:29,698 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:29,698 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:29,698 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,698 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:29,699 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:29,699 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,699 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:29,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:29,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:29,703 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:29,703 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:29,703 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:29,703 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:29,703 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:29,705 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,705 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,705 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:29,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,705 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,708 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:29,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,710 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:29,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:29,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:29,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:29,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,710 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:29,711 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:29,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:29,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:29,711 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:29,794 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/912a63f39d0f4bdc8cb5fa3f28053384 2023-04-22 19:56:29,798 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/088baf6b8a1c451691b3b0f9c7af30ec as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/088baf6b8a1c451691b3b0f9c7af30ec 2023-04-22 19:56:29,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/088baf6b8a1c451691b3b0f9c7af30ec, entries=200, sequenceid=196, filesize=32.1 K 2023-04-22 19:56:29,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/0c420fb0d2e848c487a4b06342865ee5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0c420fb0d2e848c487a4b06342865ee5 2023-04-22 19:56:29,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0c420fb0d2e848c487a4b06342865ee5, entries=150, sequenceid=196, filesize=11.8 K 2023-04-22 19:56:29,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/912a63f39d0f4bdc8cb5fa3f28053384 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/912a63f39d0f4bdc8cb5fa3f28053384 2023-04-22 19:56:29,809 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/912a63f39d0f4bdc8cb5fa3f28053384, entries=150, sequenceid=196, filesize=11.8 K 2023-04-22 19:56:29,810 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for e8d85766c3fe2800a849fb8577794261 in 862ms, sequenceid=196, compaction requested=true 2023-04-22 19:56:29,810 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:29,810 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:29,810 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:56:29,810 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:56:29,810 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:29,810 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:29,811 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:29,811 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:56:29,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:29,811 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/B is initiating minor compaction (all files) 2023-04-22 19:56:29,811 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/B in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,812 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/a689097420c54720add9e01c901a0685, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/f1ff711ed6914a829afff5541ed92c13, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/3bb364ccb15f41de96c002b4cfbf3948, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0c420fb0d2e848c487a4b06342865ee5] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=47.1 K 2023-04-22 19:56:29,812 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:56:29,812 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/A is initiating minor compaction (all files) 2023-04-22 19:56:29,812 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/A in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,812 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/a984a31c437740769b39a6e4d6611017, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/45be71951bde41118df6a1182c18fb66, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/63dc634ff8794a288cb4f157bc667745, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/088baf6b8a1c451691b3b0f9c7af30ec] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=108.1 K 2023-04-22 19:56:29,812 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a689097420c54720add9e01c901a0685, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682193386129 2023-04-22 19:56:29,813 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting a984a31c437740769b39a6e4d6611017, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682193386129 2023-04-22 19:56:29,813 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f1ff711ed6914a829afff5541ed92c13, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682193386175 2023-04-22 19:56:29,813 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 45be71951bde41118df6a1182c18fb66, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682193386175 2023-04-22 19:56:29,813 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 63dc634ff8794a288cb4f157bc667745, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682193387284 2023-04-22 19:56:29,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:29,814 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 3bb364ccb15f41de96c002b4cfbf3948, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682193387284 2023-04-22 19:56:29,814 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 088baf6b8a1c451691b3b0f9c7af30ec, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193388350 2023-04-22 19:56:29,814 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 0c420fb0d2e848c487a4b06342865ee5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193388360 2023-04-22 19:56:29,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:29,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:29,815 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:29,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:29,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:29,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,817 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:29,817 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:29,817 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:29,817 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:29,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:29,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:29,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:29,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:29,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:29,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:29,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:29,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:29,819 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:29,820 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:29,820 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:29,820 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:29,820 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:29,820 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:56:29,820 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:29,820 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:29,820 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:29,821 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:29,821 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:29,821 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:29,829 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#B#compaction#513 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:29,842 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#A#compaction#512 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:30,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:30,097 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:30,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193390116, max timestamp=1682193390179Num uniques 859; 2023-04-22 19:56:30,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193390116, max timestamp=1682193390179Num uniques 859; 2023-04-22 19:56:30,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193390116, max timestamp=1682193390179Num uniques 859; 2023-04-22 19:56:30,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:30,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:30,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:30,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:30,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50646 deadline: 1682193450240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:30,253 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/001f8a6c3e9a40b589597e645866a870 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/001f8a6c3e9a40b589597e645866a870 2023-04-22 19:56:30,257 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8d85766c3fe2800a849fb8577794261/B of e8d85766c3fe2800a849fb8577794261 into 001f8a6c3e9a40b589597e645866a870(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:30,257 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:30,257 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/B, priority=12, startTime=1682193389810; duration=0sec 2023-04-22 19:56:30,257 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:30,257 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:56:30,258 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:56:30,258 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/C is initiating minor compaction (all files) 2023-04-22 19:56:30,259 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/C in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:30,259 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/7bc4b56bad464616b4a92575f8be33f4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e6f1e7a5a70246a19df914a682e5ff7a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/070dfbbd4afb4a21815004a6de27f1a7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/912a63f39d0f4bdc8cb5fa3f28053384] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=47.1 K 2023-04-22 19:56:30,259 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 7bc4b56bad464616b4a92575f8be33f4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682193386129 2023-04-22 19:56:30,259 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting e6f1e7a5a70246a19df914a682e5ff7a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682193386175 2023-04-22 19:56:30,259 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 070dfbbd4afb4a21815004a6de27f1a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682193387284 2023-04-22 19:56:30,260 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 912a63f39d0f4bdc8cb5fa3f28053384, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193388360 2023-04-22 19:56:30,266 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#C#compaction#515 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:30,281 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422cfb7483d1a88493581c1aba8c8b715cf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cfb7483d1a88493581c1aba8c8b715cf 2023-04-22 19:56:30,291 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/0907a6bb931c4595acd165baa3651e56 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/0907a6bb931c4595acd165baa3651e56 2023-04-22 19:56:30,296 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8d85766c3fe2800a849fb8577794261/A of e8d85766c3fe2800a849fb8577794261 into 0907a6bb931c4595acd165baa3651e56(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:30,296 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:30,296 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/A, priority=12, startTime=1682193389810; duration=0sec 2023-04-22 19:56:30,296 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:30,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:30,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193450344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:30,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:30,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193450488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:30,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:30,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193450489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:30,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:30,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193450547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:30,674 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/01d7bee271e84e7ea4b03e5baeef8dce as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/01d7bee271e84e7ea4b03e5baeef8dce 2023-04-22 19:56:30,679 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8d85766c3fe2800a849fb8577794261/C of e8d85766c3fe2800a849fb8577794261 into 01d7bee271e84e7ea4b03e5baeef8dce(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:30,679 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:30,679 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/C, priority=12, startTime=1682193389810; duration=0sec 2023-04-22 19:56:30,679 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:30,686 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/24f9b879298d4d4ebce63c2d7d1b509f 2023-04-22 19:56:30,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:30,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193450851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,107 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/e2234b0ad21f4d068a7a6f66817c1b8b 2023-04-22 19:56:31,118 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/a076067d9fae4b5db3fc617c71fbaa75 2023-04-22 19:56:31,122 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/24f9b879298d4d4ebce63c2d7d1b509f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/24f9b879298d4d4ebce63c2d7d1b509f 2023-04-22 19:56:31,126 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/24f9b879298d4d4ebce63c2d7d1b509f, entries=150, sequenceid=206, filesize=25.3 K 2023-04-22 19:56:31,126 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/e2234b0ad21f4d068a7a6f66817c1b8b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e2234b0ad21f4d068a7a6f66817c1b8b 2023-04-22 19:56:31,130 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e2234b0ad21f4d068a7a6f66817c1b8b, entries=150, sequenceid=206, filesize=11.8 K 2023-04-22 19:56:31,131 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/a076067d9fae4b5db3fc617c71fbaa75 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/a076067d9fae4b5db3fc617c71fbaa75 2023-04-22 19:56:31,134 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/a076067d9fae4b5db3fc617c71fbaa75, entries=150, sequenceid=206, filesize=11.8 K 2023-04-22 19:56:31,135 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e8d85766c3fe2800a849fb8577794261 in 1315ms, sequenceid=206, compaction requested=false 2023-04-22 19:56:31,135 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:31,135 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:31,135 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:31,135 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:31,135 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,135 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:31,135 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:31,136 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:31,137 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:31,137 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a239a52[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:31,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:31,138 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:31,142 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:31,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:31,143 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,143 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:31,145 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:31,147 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:31,148 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:31,148 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:31,148 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:31,148 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:31,149 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:31,149 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:31,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:31,149 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:31,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:31,150 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,151 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,154 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,154 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:31,154 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:31,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:31,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:31,154 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:31,154 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,154 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:31,154 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:31,154 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:31,154 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:31,154 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:31,154 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:31,154 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:31,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:31,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:31,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:31,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:31,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:31,254 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:31,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:31,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:31,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:31,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:31,258 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:31,258 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:31,258 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:31,258 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:31,258 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:31,258 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:31,259 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:31,259 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:31,259 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,260 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:31,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:31,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:31,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:31,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:31,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:31,261 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:31,261 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-22 19:56:31,262 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:31,262 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:31,262 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:31,262 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:31,262 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:31,262 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:31,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:31,361 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:31,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:31,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193451379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:31,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193451445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:31,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193451446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,450 DEBUG [Thread-3621] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:56:31,451 DEBUG [Thread-3619] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:56:31,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:31,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193451482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,681 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422fa947b0ae77640b287a1d5c1804dcde2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422fa947b0ae77640b287a1d5c1804dcde2 2023-04-22 19:56:31,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:31,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193451686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:31,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:31,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193451990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,087 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=246, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/b6f2588587b143a3a1cd37553018437c 2023-04-22 19:56:32,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:32,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193452498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193452502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193452505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,506 DEBUG [Thread-3617] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:56:32,508 DEBUG [Thread-3615] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:56:32,508 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/cfa31a96ce7d43ba8bc14e72d76c45c9 2023-04-22 19:56:32,526 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/ff1ab19d535c4356b214fcb89cd6b6d1 2023-04-22 19:56:32,530 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/b6f2588587b143a3a1cd37553018437c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/b6f2588587b143a3a1cd37553018437c 2023-04-22 19:56:32,534 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/b6f2588587b143a3a1cd37553018437c, entries=150, sequenceid=246, filesize=25.3 K 2023-04-22 19:56:32,534 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-22 19:56:32,535 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/cfa31a96ce7d43ba8bc14e72d76c45c9 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/cfa31a96ce7d43ba8bc14e72d76c45c9 2023-04-22 19:56:32,538 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/cfa31a96ce7d43ba8bc14e72d76c45c9, entries=150, sequenceid=246, filesize=11.8 K 2023-04-22 19:56:32,540 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/ff1ab19d535c4356b214fcb89cd6b6d1 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ff1ab19d535c4356b214fcb89cd6b6d1 2023-04-22 19:56:32,544 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ff1ab19d535c4356b214fcb89cd6b6d1, entries=150, sequenceid=246, filesize=11.8 K 2023-04-22 19:56:32,545 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e8d85766c3fe2800a849fb8577794261 in 1284ms, sequenceid=246, compaction requested=true 2023-04-22 19:56:32,545 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:32,545 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:32,545 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:32,545 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:32,545 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,545 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:32,545 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:32,547 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:32,547 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:32,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@300f66b8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:32,550 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:32,550 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:32,550 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:32,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:32,551 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,551 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:32,552 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:32,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:32,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:32,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:32,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:32,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:32,556 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:32,556 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:32,556 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:32,556 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:32,556 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:32,558 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:32,558 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:32,558 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:32,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:32,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:32,559 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:32,560 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:32,569 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:32,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:32,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:32,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:32,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:32,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:32,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:32,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:32,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:32,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,569 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:32,569 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:32,569 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:32,570 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:32,570 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:32,670 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:32,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:32,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:32,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:32,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:32,676 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:32,676 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:32,676 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:32,676 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:32,676 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:32,676 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:32,677 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:32,677 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:32,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:32,678 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:32,678 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:32,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:32,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:32,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:32,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:32,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:32,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:32,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:32,680 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:56:32,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:32,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:32,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:32,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:32,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:32,680 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:33,105 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304222828521e85824534a3e982baf59551b9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304222828521e85824534a3e982baf59551b9 2023-04-22 19:56:33,111 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=256, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/9d3c8766a80e4c86a5e04f8585716025 2023-04-22 19:56:33,120 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/e21b43fdfe964e81b811189d7dec10d2 2023-04-22 19:56:33,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:33,503 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:33,530 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/02fd166ad91c4c21aac1d18450e9ed70 2023-04-22 19:56:33,536 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/9d3c8766a80e4c86a5e04f8585716025 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/9d3c8766a80e4c86a5e04f8585716025 2023-04-22 19:56:33,542 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/9d3c8766a80e4c86a5e04f8585716025, entries=150, sequenceid=256, filesize=25.3 K 2023-04-22 19:56:33,542 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/e21b43fdfe964e81b811189d7dec10d2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e21b43fdfe964e81b811189d7dec10d2 2023-04-22 19:56:33,546 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e21b43fdfe964e81b811189d7dec10d2, entries=150, sequenceid=256, filesize=11.8 K 2023-04-22 19:56:33,546 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/02fd166ad91c4c21aac1d18450e9ed70 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/02fd166ad91c4c21aac1d18450e9ed70 2023-04-22 19:56:33,550 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/02fd166ad91c4c21aac1d18450e9ed70, entries=150, sequenceid=256, filesize=11.8 K 2023-04-22 19:56:33,550 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=87.22 KB/89310 for e8d85766c3fe2800a849fb8577794261 in 870ms, sequenceid=256, compaction requested=true 2023-04-22 19:56:33,551 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:33,551 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:33,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:33,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:33,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:33,551 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:33,552 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,552 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,553 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:33,553 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:33,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-22 19:56:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:33,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:33,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:33,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:33,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:33,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d55415[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:33,554 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:33,554 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,555 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,556 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:33,556 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:33,556 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:33,556 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,556 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:33,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:33,558 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,558 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:33,559 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:33,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:33,562 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:33,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:33,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:33,562 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:33,562 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:33,564 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,564 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:33,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:33,564 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,568 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:33,568 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:33,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:33,568 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:33,568 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:33,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:33,568 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,568 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,568 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,568 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,568 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,568 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,568 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,568 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:33,568 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:33,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:33,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:33,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193453597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,669 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:33,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:33,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:33,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:33,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:33,673 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:33,673 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:33,673 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:33,673 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:33,673 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,673 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,674 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:33,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:33,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:33,674 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,675 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,675 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:33,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:33,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:33,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:33,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:33,676 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:33,676 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:33,676 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:33,676 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:33,676 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:33,676 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:33,676 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:33,676 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:33,677 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:33,677 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:33,677 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:33,677 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,677 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:33,677 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:33,680 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:33,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e8ed281[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:33,682 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:33,682 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,684 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:33,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:33,684 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,684 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:33,684 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:33,685 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,685 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:33,685 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,685 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:33,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:33,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:33,689 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:33,689 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:33,689 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:33,689 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:33,689 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:33,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:33,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,694 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,695 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,697 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:33,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:33,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:33,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:33,700 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:33,700 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:33,700 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:33,700 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:33,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:33,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:33,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193453702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,800 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:33,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:33,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:33,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:33,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:33,804 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:33,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:33,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:33,804 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:33,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,805 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,806 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,806 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:33,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:33,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:33,806 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,806 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:33,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:33,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:33,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:33,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:33,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:33,807 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:33,808 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:33,808 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:33,808 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:33,808 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:33,808 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:33,808 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:33,808 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:33,808 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:33,808 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,808 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:33,808 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:33,811 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:33,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48a92a08[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:33,813 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:33,813 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,814 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:33,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:33,814 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:33,814 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,814 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:33,815 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,815 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:33,815 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:33,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:33,818 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:33,818 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:33,818 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:33,818 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:33,818 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:33,820 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,820 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,820 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:33,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,821 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,822 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,825 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,825 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,825 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:33,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:33,825 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:33,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:33,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:33,825 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,825 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,825 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,825 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,825 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,825 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:33,825 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:33,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:33,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:33,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:33,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:33,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193453906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,925 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:33,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:33,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:33,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:33,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:33,929 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:33,929 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:33,929 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:33,929 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:33,929 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,929 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,930 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,930 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:33,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:33,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:33,931 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,931 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:33,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:33,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:33,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:33,932 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:33,932 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:33,932 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:33,933 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:33,933 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:33,933 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:33,933 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:33,933 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:33,933 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:33,933 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:33,933 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:33,933 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,933 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:33,933 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:33,934 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,936 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:33,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:33,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d8af03[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:33,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:33,939 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,941 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:33,942 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,942 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:33,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:33,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:33,944 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:33,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:33,946 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:33,946 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:33,946 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:33,946 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:33,947 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:33,948 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:33,949 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,949 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:33,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:33,949 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:33,950 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,953 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,953 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:33,953 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:33,953 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:33,953 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:33,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:33,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:33,953 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:33,953 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:33,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:33,953 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:33,953 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:33,953 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,954 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:33,954 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:33,974 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422c1b68ad177944f36984cf28e1ce7e99e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c1b68ad177944f36984cf28e1ce7e99e 2023-04-22 19:56:34,054 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:34,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:34,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:34,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:34,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:34,057 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:34,057 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,057 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:34,057 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:34,058 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,058 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,059 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,059 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,059 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,059 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:34,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:34,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:34,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:34,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:34,060 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:34,060 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,060 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:34,060 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:34,061 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:34,061 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,061 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:34,061 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:34,061 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,061 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:34,061 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:34,061 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,061 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:34,061 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:34,064 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,064 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,064 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,064 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:34,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b853691[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:34,067 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:34,067 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,068 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:34,068 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,069 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:34,069 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,069 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,069 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,070 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:34,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:34,072 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:34,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:34,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:34,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:34,072 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:34,073 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,073 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,074 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,074 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,075 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:34,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:34,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,082 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,081 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,082 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:34,082 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:34,082 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,082 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,082 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,082 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,082 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,182 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:34,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:34,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:34,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:34,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:34,185 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:34,185 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,185 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:34,185 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:34,186 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,186 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,187 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:34,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:34,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:34,188 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:34,188 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,188 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:34,189 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:34,189 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:34,189 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,189 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:34,189 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:34,189 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,189 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:34,189 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:34,189 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,189 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:34,189 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:34,190 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,190 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,191 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:34,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43f7e3ba[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:34,192 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:34,192 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,194 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,194 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:34,194 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:34,194 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:34,194 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,194 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:34,195 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:34,195 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:34,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:34,198 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:34,198 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:34,198 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:34,198 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:34,198 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:34,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,200 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,201 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,203 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,203 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,203 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,203 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,203 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:34,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:34,203 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,203 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,203 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,204 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,204 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:34,204 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:34,203 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:34,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193454212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,304 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:34,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:34,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:34,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:34,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:34,307 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:34,307 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,307 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:34,307 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:34,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,308 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,310 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,310 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,311 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,311 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:34,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:34,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:34,312 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:34,312 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,312 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:34,312 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:34,312 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:34,312 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,313 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:34,313 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:34,313 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,313 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:34,313 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:34,313 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,313 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:34,313 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:34,314 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,315 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,315 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:34,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@428f68a6[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:34,317 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:34,317 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,318 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,318 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:34,318 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:34,318 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:34,318 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,318 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:34,319 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,319 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:34,319 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,319 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:34,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:34,322 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:34,322 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:34,322 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:34,322 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:34,322 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:34,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,325 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,325 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,326 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,329 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,329 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,330 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:34,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:34,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:34,330 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:34,330 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:34,380 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=273, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/c0de3ef26346424fb879d916b3597aed 2023-04-22 19:56:34,430 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:34,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:34,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:34,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:34,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:34,434 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:34,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:34,434 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:34,435 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,435 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,436 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,437 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:34,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:34,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:34,438 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,438 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:34,438 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:34,438 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,439 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:34,439 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:34,440 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,440 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,441 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,441 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:34,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:34,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4697743[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:34,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:34,443 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,445 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,445 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:34,445 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:34,445 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:34,445 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,446 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:34,447 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:34,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:34,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:34,451 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:34,451 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:34,451 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:34,451 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:34,451 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:34,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,454 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,456 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,461 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,461 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,461 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,461 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,461 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:34,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,462 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:34,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:34,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,462 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:34,462 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:34,462 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,562 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:34,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:34,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:34,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:34,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:34,566 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:34,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:34,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:34,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,566 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,567 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,567 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,568 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,568 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:34,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:34,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:34,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:34,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:34,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:34,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:34,569 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,569 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:34,569 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:34,569 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:34,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:34,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:34,569 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:34,571 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,571 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,571 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,571 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:34,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:34,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6daf8e1[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:34,572 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:34,572 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,575 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:34,575 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,575 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:34,576 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:34,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:34,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:34,578 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:34,578 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:34,578 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:34,578 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:34,578 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:34,580 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,580 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,580 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,580 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,581 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:34,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:34,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,584 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,584 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:34,584 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:34,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,684 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:34,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:34,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:34,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:34,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:34,688 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:34,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:34,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:34,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,688 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,691 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,691 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,691 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,691 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:34,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:34,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:34,692 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:34,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:34,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:34,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:34,693 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,693 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:34,693 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:34,693 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:34,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:34,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:34,693 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:34,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,695 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,695 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:34,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a4c31b3[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:34,697 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:34,697 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,698 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,698 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:34,698 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:34,698 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:34,698 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,698 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:34,699 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:34,699 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,699 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,699 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:34,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:34,703 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:34,703 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:34,703 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:34,703 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:34,703 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:34,705 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,705 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,705 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,706 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,707 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,709 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,709 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,710 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:34,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:34,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,710 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,710 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:34,710 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:34,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:34,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193454716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/58c79b7d406c46d38128bd40935b2d30 2023-04-22 19:56:34,810 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:34,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:34,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:34,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:34,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:34,813 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:34,813 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,813 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:34,813 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:34,814 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,814 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,815 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,815 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:34,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:34,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:34,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:34,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:34,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:34,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:34,816 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,816 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:34,816 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:34,816 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:34,816 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:34,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:34,817 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:34,818 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,818 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,818 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,818 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:34,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43bfbc04[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:34,819 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:34,819 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,821 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,821 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:34,821 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,821 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:34,821 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:34,822 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:34,823 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:34,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:34,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:34,825 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:34,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:34,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:34,825 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:34,825 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:34,826 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,826 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,826 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,827 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,828 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,830 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,830 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,830 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,830 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:34,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:34,830 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:34,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,830 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,830 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,830 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:34,831 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,831 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:34,831 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,831 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,931 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:34,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:34,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:34,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:34,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:34,934 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:34,934 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,934 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:34,934 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:34,934 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,934 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,936 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,936 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,937 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:34,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:34,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:34,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:34,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:34,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:34,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:34,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:34,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:34,938 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,938 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:34,938 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:34,938 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:34,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:34,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:34,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:34,938 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:34,942 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,942 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:34,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:34,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c23185f[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:34,943 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:34,943 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,944 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,944 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:34,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:34,945 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,945 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:34,945 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:34,946 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:34,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,946 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:34,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:34,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:34,948 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:34,948 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:34,948 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:34,948 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:34,948 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:34,955 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,955 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,955 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:34,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,956 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:34,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,964 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:34,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:34,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:34,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:34,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:34,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:34,965 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:34,966 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:34,965 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:34,966 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:34,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:34,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:34,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:34,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:35,066 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:35,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:35,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:35,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:35,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:35,070 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:35,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:35,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:35,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:35,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,070 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,071 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:35,071 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:35,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,072 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:35,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:35,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:35,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:35,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:35,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:35,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:35,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:35,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:35,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:35,073 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:35,073 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:35,073 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:35,073 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:35,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:35,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:35,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:35,073 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:35,075 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:35,075 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,075 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,075 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:35,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:35,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:35,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d6fe4ca[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:35,077 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:35,077 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,079 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,079 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:35,079 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:35,079 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:35,079 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,079 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:35,080 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,080 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:35,080 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:35,080 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:35,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:35,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:35,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:35,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:35,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:35,083 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:35,083 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:35,083 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:35,083 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:35,083 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:35,084 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,084 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,084 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:35,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:35,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:35,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:35,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:35,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:35,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,086 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,087 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:35,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:35,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:35,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,090 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,090 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:35,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:35,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:35,090 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:35,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:35,091 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:35,191 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:35,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:35,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:35,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:35,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:35,194 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:35,194 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:35,194 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:35,194 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:35,194 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,195 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,196 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:35,197 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:35,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:35,197 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,197 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:35,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:35,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:35,198 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/095b8929cc4b4fa19d1ea6d3d4d0386e 2023-04-22 19:56:35,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:35,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:35,198 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:35,198 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:35,198 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:35,198 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:35,198 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:35,198 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:35,199 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:35,199 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:35,199 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:35,199 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:35,199 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:35,199 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,199 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:35,199 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:35,200 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,200 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:35,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:35,201 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,201 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/c0de3ef26346424fb879d916b3597aed as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c0de3ef26346424fb879d916b3597aed 2023-04-22 19:56:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:35,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:35,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4635977d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:35,203 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:35,203 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,204 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:35,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:35,204 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:35,204 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,204 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:35,205 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,205 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:35,205 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:35,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:35,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:35,206 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:35,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,207 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c0de3ef26346424fb879d916b3597aed, entries=200, sequenceid=273, filesize=32.3 K 2023-04-22 19:56:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:35,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/58c79b7d406c46d38128bd40935b2d30 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/58c79b7d406c46d38128bd40935b2d30 2023-04-22 19:56:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:35,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:35,209 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:35,209 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:35,209 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:35,209 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:35,209 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:35,211 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/58c79b7d406c46d38128bd40935b2d30, entries=150, sequenceid=273, filesize=11.9 K 2023-04-22 19:56:35,211 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,211 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,211 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:35,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/095b8929cc4b4fa19d1ea6d3d4d0386e as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/095b8929cc4b4fa19d1ea6d3d4d0386e 2023-04-22 19:56:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:35,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:35,212 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,213 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:35,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/095b8929cc4b4fa19d1ea6d3d4d0386e, entries=150, sequenceid=273, filesize=11.9 K 2023-04-22 19:56:35,216 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,216 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:35,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:35,216 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,216 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,216 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:35,216 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for e8d85766c3fe2800a849fb8577794261 in 1663ms, sequenceid=273, compaction requested=true 2023-04-22 19:56:35,216 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:35,216 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:35,216 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:35,216 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,216 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:35,216 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:35,217 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:35,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:35,217 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:56:35,216 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:35,217 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:35,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:35,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:35,217 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:56:35,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:35,218 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60706 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:56:35,218 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/B is initiating minor compaction (all files) 2023-04-22 19:56:35,218 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/B in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:35,218 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/001f8a6c3e9a40b589597e645866a870, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e2234b0ad21f4d068a7a6f66817c1b8b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/cfa31a96ce7d43ba8bc14e72d76c45c9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e21b43fdfe964e81b811189d7dec10d2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/58c79b7d406c46d38128bd40935b2d30] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=59.3 K 2023-04-22 19:56:35,218 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137046 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:56:35,218 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/A is initiating minor compaction (all files) 2023-04-22 19:56:35,219 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/A in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:35,219 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/0907a6bb931c4595acd165baa3651e56, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/24f9b879298d4d4ebce63c2d7d1b509f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/b6f2588587b143a3a1cd37553018437c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/9d3c8766a80e4c86a5e04f8585716025, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c0de3ef26346424fb879d916b3597aed] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=133.8 K 2023-04-22 19:56:35,219 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 001f8a6c3e9a40b589597e645866a870, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193388360 2023-04-22 19:56:35,219 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 0907a6bb931c4595acd165baa3651e56, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193388360 2023-04-22 19:56:35,219 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting e2234b0ad21f4d068a7a6f66817c1b8b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682193388956 2023-04-22 19:56:35,220 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 24f9b879298d4d4ebce63c2d7d1b509f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682193388956 2023-04-22 19:56:35,220 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting cfa31a96ce7d43ba8bc14e72d76c45c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682193390222 2023-04-22 19:56:35,220 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting b6f2588587b143a3a1cd37553018437c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682193390222 2023-04-22 19:56:35,220 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting e21b43fdfe964e81b811189d7dec10d2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1682193391365 2023-04-22 19:56:35,220 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 9d3c8766a80e4c86a5e04f8585716025, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1682193391365 2023-04-22 19:56:35,220 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 58c79b7d406c46d38128bd40935b2d30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682193393531 2023-04-22 19:56:35,220 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting c0de3ef26346424fb879d916b3597aed, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682193393531 2023-04-22 19:56:35,231 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#B#compaction#528 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:35,232 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#A#compaction#527 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:35,258 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/1a00f9e089704d2ca006c1a72b6ef871 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/1a00f9e089704d2ca006c1a72b6ef871 2023-04-22 19:56:35,263 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e8d85766c3fe2800a849fb8577794261/B of e8d85766c3fe2800a849fb8577794261 into 1a00f9e089704d2ca006c1a72b6ef871(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:35,263 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:35,263 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/B, priority=11, startTime=1682193395217; duration=0sec 2023-04-22 19:56:35,263 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:35,264 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 19:56:35,266 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60706 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 19:56:35,266 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/C is initiating minor compaction (all files) 2023-04-22 19:56:35,267 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/C in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:35,267 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/01d7bee271e84e7ea4b03e5baeef8dce, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/a076067d9fae4b5db3fc617c71fbaa75, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ff1ab19d535c4356b214fcb89cd6b6d1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/02fd166ad91c4c21aac1d18450e9ed70, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/095b8929cc4b4fa19d1ea6d3d4d0386e] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=59.3 K 2023-04-22 19:56:35,267 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 01d7bee271e84e7ea4b03e5baeef8dce, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682193388360 2023-04-22 19:56:35,267 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a076067d9fae4b5db3fc617c71fbaa75, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682193388956 2023-04-22 19:56:35,267 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ff1ab19d535c4356b214fcb89cd6b6d1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682193390222 2023-04-22 19:56:35,268 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 02fd166ad91c4c21aac1d18450e9ed70, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1682193391365 2023-04-22 19:56:35,268 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 095b8929cc4b4fa19d1ea6d3d4d0386e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682193393531 2023-04-22 19:56:35,275 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/3112c90c460540b68ce3c5535839f18c as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/3112c90c460540b68ce3c5535839f18c 2023-04-22 19:56:35,277 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#C#compaction#529 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:35,280 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e8d85766c3fe2800a849fb8577794261/A of e8d85766c3fe2800a849fb8577794261 into 3112c90c460540b68ce3c5535839f18c(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:35,281 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:35,281 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/A, priority=11, startTime=1682193395216; duration=0sec 2023-04-22 19:56:35,281 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:35,317 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:35,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:35,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:35,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:35,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:35,320 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:35,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:35,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:35,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:35,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,320 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,321 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,321 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:35,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:35,322 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,322 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:35,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:35,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:35,323 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:35,323 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:35,324 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 19:56:35,324 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:35,324 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:35,324 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:35,324 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:35,324 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:35,324 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:35,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:35,456 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:35,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:35,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193455479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:35,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193455480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:35,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193455584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:35,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193455585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,689 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/f9275824e4f6480c9ab353a190a8ac50 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f9275824e4f6480c9ab353a190a8ac50 2023-04-22 19:56:35,693 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e8d85766c3fe2800a849fb8577794261/C of e8d85766c3fe2800a849fb8577794261 into f9275824e4f6480c9ab353a190a8ac50(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:35,693 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:35,694 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/C, priority=11, startTime=1682193395217; duration=0sec 2023-04-22 19:56:35,694 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:35,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:35,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193455723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,755 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042264a77d8f65ad42729bb9d24755edc661 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042264a77d8f65ad42729bb9d24755edc661 2023-04-22 19:56:35,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:35,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193455785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:35,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:35,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193455789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:36,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193456090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:36,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193456093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,161 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/e1f29bd0de8a4054889fb258a4f2f670 2023-04-22 19:56:36,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:36,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50640 deadline: 1682193456515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,518 DEBUG [Thread-3615] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:56:36,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:36,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50652 deadline: 1682193456536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,541 DEBUG [Thread-3617] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:56:36,574 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/68a4b2a8f9ec41889648588d5a5bc460 2023-04-22 19:56:36,584 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/2837ed608a014ae7978801e17b099664 2023-04-22 19:56:36,588 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/e1f29bd0de8a4054889fb258a4f2f670 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/e1f29bd0de8a4054889fb258a4f2f670 2023-04-22 19:56:36,593 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/e1f29bd0de8a4054889fb258a4f2f670, entries=150, sequenceid=290, filesize=25.4 K 2023-04-22 19:56:36,594 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/68a4b2a8f9ec41889648588d5a5bc460 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/68a4b2a8f9ec41889648588d5a5bc460 2023-04-22 19:56:36,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:36,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193456594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,599 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/68a4b2a8f9ec41889648588d5a5bc460, entries=150, sequenceid=290, filesize=11.9 K 2023-04-22 19:56:36,600 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/2837ed608a014ae7978801e17b099664 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/2837ed608a014ae7978801e17b099664 2023-04-22 19:56:36,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:36,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193456599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,604 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/2837ed608a014ae7978801e17b099664, entries=150, sequenceid=290, filesize=11.9 K 2023-04-22 19:56:36,605 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e8d85766c3fe2800a849fb8577794261 in 1282ms, sequenceid=290, compaction requested=false 2023-04-22 19:56:36,605 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:36,605 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:36,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:36,609 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,609 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:36,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:36,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:36,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:36,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78545c23[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:36,611 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:36,611 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:36,615 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:36,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:36,616 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:36,616 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:36,616 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:36,616 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,616 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:36,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:36,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:36,617 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,617 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:36,620 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:36,621 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:36,621 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:36,621 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:36,621 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:36,622 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:36,622 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:36,622 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:36,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:36,622 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:36,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:36,624 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:36,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:36,628 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:36,628 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:36,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:36,628 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:36,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:36,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:36,628 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:36,628 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:36,628 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:36,629 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,629 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:36,629 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:36,629 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:36,629 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:36,629 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:36,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:36,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:36,729 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:36,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:36,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:36,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:36,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:36,732 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:36,732 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:36,733 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:36,733 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:36,733 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:36,733 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:36,734 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:36,734 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:36,734 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:36,734 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:36,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:36,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:36,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:36,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:36,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:36,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:36,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:36,737 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:36,737 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:36,737 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:36,737 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:36,737 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-22 19:56:36,737 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:36,738 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:36,738 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:36,738 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:36,738 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:36,738 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:37,153 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042229db8b866c404fab9d445091fa973ec1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042229db8b866c404fab9d445091fa973ec1 2023-04-22 19:56:37,159 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=308, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/ba3e4c632c534426b7686c878182d292 2023-04-22 19:56:37,169 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/a27818f710b740ff9cb85be9512992c2 2023-04-22 19:56:37,179 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/e1bb3c0a5954441f83c79ff499af70cc 2023-04-22 19:56:37,183 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/ba3e4c632c534426b7686c878182d292 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/ba3e4c632c534426b7686c878182d292 2023-04-22 19:56:37,187 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/ba3e4c632c534426b7686c878182d292, entries=150, sequenceid=308, filesize=25.4 K 2023-04-22 19:56:37,187 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/a27818f710b740ff9cb85be9512992c2 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/a27818f710b740ff9cb85be9512992c2 2023-04-22 19:56:37,191 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/a27818f710b740ff9cb85be9512992c2, entries=150, sequenceid=308, filesize=11.9 K 2023-04-22 19:56:37,191 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/e1bb3c0a5954441f83c79ff499af70cc as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e1bb3c0a5954441f83c79ff499af70cc 2023-04-22 19:56:37,194 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e1bb3c0a5954441f83c79ff499af70cc, entries=150, sequenceid=308, filesize=11.9 K 2023-04-22 19:56:37,195 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for e8d85766c3fe2800a849fb8577794261 in 458ms, sequenceid=308, compaction requested=true 2023-04-22 19:56:37,195 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:37,195 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,195 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:37,195 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:37,195 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,195 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:37,195 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:37,197 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,197 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,197 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,197 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:37,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c215296[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:37,198 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:37,199 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,199 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,200 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:37,200 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:37,200 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:37,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,200 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:37,202 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,202 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:37,202 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,202 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:37,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:37,205 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:37,205 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:37,205 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:37,205 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:37,205 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:37,207 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,207 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,207 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,208 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,209 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,211 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,211 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:37,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:37,211 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,211 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,212 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,212 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,212 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,212 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,212 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:37,212 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,212 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:37,212 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:37,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,312 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:37,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:37,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:37,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:37,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:37,316 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:37,316 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,316 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:37,316 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:37,316 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,317 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,318 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,318 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,318 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,318 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:37,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:37,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:37,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:37,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:37,319 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:37,319 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,319 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:37,320 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,320 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:37,320 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:37,320 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:37,323 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,323 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:37,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ecab26c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:37,325 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:37,325 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,326 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,326 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:37,327 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,327 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:37,327 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:37,327 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:37,328 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,328 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:37,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:37,331 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:37,331 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:37,331 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:37,331 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:37,331 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:37,333 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,333 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,333 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,336 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,337 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,340 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,340 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,340 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,340 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:37,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:37,340 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,341 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,341 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,341 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,341 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,340 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:37,341 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:37,341 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:37,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,441 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:37,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:37,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:37,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:37,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:37,445 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:37,445 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,445 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:37,445 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:37,445 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,445 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,446 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,446 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,447 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:37,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:37,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:37,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:37,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:37,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:37,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:37,448 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:37,449 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,449 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:37,449 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:37,449 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:37,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:37,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:37,449 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:37,450 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,450 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,451 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,451 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:37,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24ae4ab7[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:37,452 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:37,453 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,453 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:37,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:37,454 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:37,454 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,454 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:37,457 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:37,457 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,457 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,457 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:37,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:37,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:37,460 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:37,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:37,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:37,460 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:37,460 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:37,463 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,463 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,464 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,464 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,465 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,468 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:37,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:37,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,468 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:37,469 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:37,468 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,469 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,521 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-22 19:56:37,569 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:37,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:37,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:37,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:37,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:37,572 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:37,572 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,572 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:37,572 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:37,573 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,573 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,574 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,574 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,574 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,575 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:37,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:37,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:37,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:37,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:37,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:37,576 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,576 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:37,576 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:37,576 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:37,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:37,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,576 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:37,577 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:37,579 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,579 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,580 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,580 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:37,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ec1af45[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:37,582 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:37,582 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,583 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,583 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,583 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:37,583 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:37,583 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:37,583 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:37,584 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:37,584 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,584 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,585 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:37,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:37,587 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:37,587 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:37,587 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:37,587 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:37,587 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:37,589 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,589 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,589 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,590 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,591 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,594 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,594 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,594 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,594 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:37,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:37,594 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,595 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:37,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:37,595 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:37,595 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,595 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,595 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,595 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,595 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:37,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 19:56:37,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:37,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:37,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:37,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:37,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:37,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:37,695 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:37,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:37,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:37,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193397681, max timestamp=1682193397689Num uniques 859; 2023-04-22 19:56:37,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682193397681, max timestamp=1682193397689Num uniques 859; 2023-04-22 19:56:37,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:56:37,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193397685, max timestamp=1682193397689Num uniques 859; 2023-04-22 19:56:37,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 19:56:37,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:37,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:37,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:37,700 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:37,700 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,700 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:37,700 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:37,702 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,702 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,703 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,703 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,703 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,703 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:37,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:37,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:37,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:37,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:37,705 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:37,705 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,705 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:37,705 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:37,705 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:37,706 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,706 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:37,706 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:37,706 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,706 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:37,706 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:37,706 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,706 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:37,706 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:37,707 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,707 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,708 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,708 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:37,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1892f7e9[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:37,710 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:37,710 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,711 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,711 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:37,711 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:37,711 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:37,711 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,711 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:37,713 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,713 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:37,713 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,714 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:37,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:37,716 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:37,716 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:37,716 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:37,716 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:37,716 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:37,717 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,717 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,718 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,718 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,719 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,721 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,721 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,721 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,721 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,721 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,721 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:37,721 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,721 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,722 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:37,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:37,722 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,722 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:37,722 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:37,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:37,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193457729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:37,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193457731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:37,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193457740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,743 DEBUG [Thread-3623] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., hostname=jenkins-hbase4.apache.org,44791,1682193215766, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 19:56:37,822 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:37,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:37,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:37,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:37,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:37,826 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:37,826 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,826 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:37,826 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:37,826 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,826 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,827 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,827 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,828 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,828 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:37,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:37,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:37,829 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:37,829 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,829 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:37,829 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:37,829 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:37,829 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,830 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:37,830 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:37,830 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,830 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:37,830 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:37,830 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,830 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:37,830 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:37,833 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,833 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,833 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,833 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193457832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:37,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d06f45[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:37,834 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:37,835 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:37,836 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193457834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,836 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,836 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:37,836 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:37,836 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:37,836 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:37,838 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:37,838 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,838 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,838 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:37,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:37,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:37,840 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:37,840 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:37,840 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:37,840 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:37,840 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:37,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,842 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,843 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,845 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,860 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,860 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,860 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,860 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,860 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,860 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,860 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,860 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,860 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,860 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:37,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:37,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:37,861 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:37,861 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:37,961 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:37,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:37,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:37,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:37,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:37,965 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:37,965 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,965 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:37,965 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:37,965 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,965 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,966 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,966 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,966 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,966 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:37,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:37,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:37,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:37,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:37,968 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,968 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:37,968 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:37,968 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:37,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,970 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,970 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:37,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:37,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55fd767a[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:37,972 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:37,972 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,974 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,974 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:37,974 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:37,974 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:37,975 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,975 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:37,976 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:37,976 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,976 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,976 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:37,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:37,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:37,980 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:37,980 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:37,980 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:37,980 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:37,980 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:37,981 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,981 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,981 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:37,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:37,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:37,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:37,982 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:37,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,983 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,985 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:37,985 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:37,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:37,986 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,986 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:37,986 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:37,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:37,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:37,986 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:37,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:37,986 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:37,986 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,986 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:37,986 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:37,986 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:37,986 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:37,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:37,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:38,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:38,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193458036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:38,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193458038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,065 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304223451950209084863aeb8ea675cef67cf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223451950209084863aeb8ea675cef67cf 2023-04-22 19:56:38,086 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:38,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:38,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:38,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:38,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:38,091 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:38,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:38,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:38,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:38,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,091 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,093 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:38,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:38,093 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,094 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,094 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:38,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:38,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:38,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:38,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:38,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:38,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:38,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:38,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:38,095 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,095 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:38,095 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:38,095 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:38,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:38,095 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,096 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:38,096 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:38,097 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,097 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,097 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,097 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:38,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26354007[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:38,099 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:38,099 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,100 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,100 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,100 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:38,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:38,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:38,100 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:38,101 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,101 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:38,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:38,102 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:38,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:38,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:38,103 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:38,104 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:38,104 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:38,104 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:38,104 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:38,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:38,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,106 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,107 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,110 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:38,114 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:38,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:38,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:38,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:38,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:38,114 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:38,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:38,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:38,115 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:38,115 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:38,215 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:38,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:38,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:38,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:38,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:38,219 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:38,219 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:38,219 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:38,219 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:38,219 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,219 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,220 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:38,220 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:38,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,221 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:38,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:38,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:38,222 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,222 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:38,222 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:38,222 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:38,223 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:38,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,224 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,224 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:38,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:38,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2917d697[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:38,226 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:38,226 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,228 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:38,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:38,228 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:38,228 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,228 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:38,229 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:38,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:38,230 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:38,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:38,232 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:38,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:38,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:38,232 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:38,232 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:38,233 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,233 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,234 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,234 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,236 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,242 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,243 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:38,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:38,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:38,243 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,243 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:38,243 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:38,243 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:38,243 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:38,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:38,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:38,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:38,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:38,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193458339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:38,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193458340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,343 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:38,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:38,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:38,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:38,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:38,347 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:38,347 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:38,347 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:38,347 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:38,347 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,347 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,348 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:38,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:38,348 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,349 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:38,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:38,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:38,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:38,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:38,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:38,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:38,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:38,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:38,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:38,350 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,350 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:38,350 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:38,350 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:38,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:38,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:38,350 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:38,352 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,352 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,352 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,352 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:38,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29bd3eb7[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:38,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:38,354 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,355 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:38,355 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,355 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:38,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:38,355 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:38,357 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,357 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:38,357 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:38,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,357 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:38,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:38,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:38,360 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:38,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:38,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:38,360 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:38,360 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:38,361 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,361 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:38,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,361 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:38,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:38,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,362 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,363 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,365 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,365 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:38,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:38,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:38,366 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:38,366 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:38,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:38,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:38,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:38,366 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:38,366 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:38,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:38,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:38,466 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:38,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:38,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:38,470 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=321, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/18c56a4b77d540aa93a40dde5ab28105 2023-04-22 19:56:38,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:38,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:38,470 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:38,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:38,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:38,470 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:38,471 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,471 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,472 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:38,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:38,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:38,473 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,473 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:38,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:38,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:38,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:38,474 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:38,474 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:38,474 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:38,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:38,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:38,475 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,475 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:38,475 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:38,475 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:38,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:38,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:38,475 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:38,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,480 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,480 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,480 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:38,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:38,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174dd35c[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:38,482 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:38,482 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/056e8f8f490a456084ecbd3b2e9da10b 2023-04-22 19:56:38,483 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,484 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:38,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:38,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:38,484 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:38,485 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,486 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:38,486 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:38,486 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:38,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:38,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:38,490 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:38,490 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:38,490 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:38,490 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:38,490 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:38,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:38,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:38,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,492 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:38,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:38,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:38,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:38,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,493 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:38,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,494 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:38,495 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/f1b7d853d15b4e2fa88886c18674d466 2023-04-22 19:56:38,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:38,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:38,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:38,499 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:38,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:38,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:38,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:38,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:38,499 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:38,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:38,499 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:38,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:38,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:38,499 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:38,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/18c56a4b77d540aa93a40dde5ab28105 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/18c56a4b77d540aa93a40dde5ab28105 2023-04-22 19:56:38,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/18c56a4b77d540aa93a40dde5ab28105, entries=300, sequenceid=321, filesize=46.1 K 2023-04-22 19:56:38,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/056e8f8f490a456084ecbd3b2e9da10b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/056e8f8f490a456084ecbd3b2e9da10b 2023-04-22 19:56:38,508 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/056e8f8f490a456084ecbd3b2e9da10b, entries=100, sequenceid=321, filesize=9.5 K 2023-04-22 19:56:38,509 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/f1b7d853d15b4e2fa88886c18674d466 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f1b7d853d15b4e2fa88886c18674d466 2023-04-22 19:56:38,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f1b7d853d15b4e2fa88886c18674d466, entries=100, sequenceid=321, filesize=9.5 K 2023-04-22 19:56:38,513 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for e8d85766c3fe2800a849fb8577794261 in 860ms, sequenceid=321, compaction requested=true 2023-04-22 19:56:38,513 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:38,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:38,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:38,513 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:56:38,513 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:56:38,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:38,514 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 125850 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:56:38,515 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:56:38,515 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/A is initiating minor compaction (all files) 2023-04-22 19:56:38,515 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/B is initiating minor compaction (all files) 2023-04-22 19:56:38,515 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/A in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,515 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/3112c90c460540b68ce3c5535839f18c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/e1f29bd0de8a4054889fb258a4f2f670, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/ba3e4c632c534426b7686c878182d292, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/18c56a4b77d540aa93a40dde5ab28105] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=122.9 K 2023-04-22 19:56:38,515 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/B in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,516 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/1a00f9e089704d2ca006c1a72b6ef871, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/68a4b2a8f9ec41889648588d5a5bc460, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/a27818f710b740ff9cb85be9512992c2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/056e8f8f490a456084ecbd3b2e9da10b] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=45.7 K 2023-04-22 19:56:38,516 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 3112c90c460540b68ce3c5535839f18c, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682193393531 2023-04-22 19:56:38,516 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 1a00f9e089704d2ca006c1a72b6ef871, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682193393531 2023-04-22 19:56:38,516 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting e1f29bd0de8a4054889fb258a4f2f670, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1682193393589 2023-04-22 19:56:38,516 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 68a4b2a8f9ec41889648588d5a5bc460, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1682193393589 2023-04-22 19:56:38,516 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting ba3e4c632c534426b7686c878182d292, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1682193395475 2023-04-22 19:56:38,517 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting a27818f710b740ff9cb85be9512992c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1682193395475 2023-04-22 19:56:38,517 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 18c56a4b77d540aa93a40dde5ab28105, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682193397620 2023-04-22 19:56:38,517 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 056e8f8f490a456084ecbd3b2e9da10b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682193397647 2023-04-22 19:56:38,525 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#B#compaction#540 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:38,527 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#A#compaction#539 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:38,544 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/733a27fcda484297a575e0a76d62ba0a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/733a27fcda484297a575e0a76d62ba0a 2023-04-22 19:56:38,548 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8d85766c3fe2800a849fb8577794261/B of e8d85766c3fe2800a849fb8577794261 into 733a27fcda484297a575e0a76d62ba0a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:38,548 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:38,548 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/B, priority=12, startTime=1682193398513; duration=0sec 2023-04-22 19:56:38,548 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:38,548 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 19:56:38,549 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 19:56:38,549 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/C is initiating minor compaction (all files) 2023-04-22 19:56:38,549 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/C in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,550 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f9275824e4f6480c9ab353a190a8ac50, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/2837ed608a014ae7978801e17b099664, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e1bb3c0a5954441f83c79ff499af70cc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f1b7d853d15b4e2fa88886c18674d466] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=45.7 K 2023-04-22 19:56:38,550 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f9275824e4f6480c9ab353a190a8ac50, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682193393531 2023-04-22 19:56:38,550 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 2837ed608a014ae7978801e17b099664, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1682193393589 2023-04-22 19:56:38,550 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting e1bb3c0a5954441f83c79ff499af70cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1682193395475 2023-04-22 19:56:38,551 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting f1b7d853d15b4e2fa88886c18674d466, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682193397647 2023-04-22 19:56:38,557 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#C#compaction#541 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:38,570 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/4d5486edca834f739603c2e482c61710 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/4d5486edca834f739603c2e482c61710 2023-04-22 19:56:38,574 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8d85766c3fe2800a849fb8577794261/C of e8d85766c3fe2800a849fb8577794261 into 4d5486edca834f739603c2e482c61710(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:38,574 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:38,574 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/C, priority=12, startTime=1682193398513; duration=0sec 2023-04-22 19:56:38,574 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:38,600 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:38,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:38,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:38,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:38,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:38,603 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:38,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:38,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:38,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:38,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,603 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,604 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,604 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:38,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:38,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:38,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:38,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:38,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:38,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:38,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:38,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:38,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:38,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:38,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:38,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:38,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:38,606 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:38,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:38,606 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:38,606 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-04-22 19:56:38,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:38,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:38,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:38,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:38,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:38,607 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:38,619 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422f63448a994e340aa944c7117866556ea to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f63448a994e340aa944c7117866556ea 2023-04-22 19:56:38,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:38,848 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:38,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:38,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:38,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193458861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193458861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:38,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193458966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:38,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:38,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193458966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,028 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=359, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/93f02e12daf94a0d936d3767cdba2405 2023-04-22 19:56:39,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:39,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50664 deadline: 1682193459171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:39,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50704 deadline: 1682193459171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,438 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/4baaef097b4c4c6b968454c7e765915d 2023-04-22 19:56:39,450 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/063ff417381847a79c65869b9930a1d1 2023-04-22 19:56:39,454 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/93f02e12daf94a0d936d3767cdba2405 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/93f02e12daf94a0d936d3767cdba2405 2023-04-22 19:56:39,457 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/93f02e12daf94a0d936d3767cdba2405, entries=150, sequenceid=359, filesize=25.4 K 2023-04-22 19:56:39,458 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/4baaef097b4c4c6b968454c7e765915d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/4baaef097b4c4c6b968454c7e765915d 2023-04-22 19:56:39,461 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/4baaef097b4c4c6b968454c7e765915d, entries=150, sequenceid=359, filesize=11.9 K 2023-04-22 19:56:39,462 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/063ff417381847a79c65869b9930a1d1 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/063ff417381847a79c65869b9930a1d1 2023-04-22 19:56:39,465 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/063ff417381847a79c65869b9930a1d1, entries=150, sequenceid=359, filesize=11.9 K 2023-04-22 19:56:39,466 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=53.67 KB/54960 for e8d85766c3fe2800a849fb8577794261 in 860ms, sequenceid=359, compaction requested=false 2023-04-22 19:56:39,466 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:39,466 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:39,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:39,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:39,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:39,466 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:39,467 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,467 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:39,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf48cca[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:39,471 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:39,471 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,472 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,472 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:39,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:39,473 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:39,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,473 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:39,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:39,477 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:39,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:39,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:39,477 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:39,477 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:39,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:39,478 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 19:56:39,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:39,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:39,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:39,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:39,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:39,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:39,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:39,480 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,481 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,485 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,490 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,490 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:39,490 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:39,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:39,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:39,490 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:39,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:39,490 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:39,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:39,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:39,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:39,490 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,490 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,490 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,491 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,491 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:39,491 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:39,492 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422c215f197ebfd46c7bdc00ead024866ed to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c215f197ebfd46c7bdc00ead024866ed 2023-04-22 19:56:39,496 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/786bc68e646942af8e1399d69f4064b5 2023-04-22 19:56:39,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/54a75aee345749c08ec4605807d0bdea 2023-04-22 19:56:39,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:39,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50664 deadline: 1682193459509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:39,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50704 deadline: 1682193459510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,591 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:39,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:39,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:39,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:39,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:39,595 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:39,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:39,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:39,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:39,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,595 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,596 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,596 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:39,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:39,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:39,597 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,597 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:39,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:39,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:39,598 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:39,598 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:39,598 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:39,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:39,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:39,599 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:39,599 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:39,599 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:39,599 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:39,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:39,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:39,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:39,599 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:39,604 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,604 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,604 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,604 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:39,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:39,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc0424d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:39,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:39,605 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,607 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,607 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:39,608 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:39,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:39,608 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:39,610 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:39,610 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:39,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:39,613 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:39,613 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:39,613 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:39,613 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:39,613 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:39,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,614 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:39,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:39,615 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:39,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50664 deadline: 1682193459613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,617 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:39,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50704 deadline: 1682193459616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,620 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:39,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:39,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:39,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,620 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:39,621 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:39,621 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:39,621 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:39,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:39,621 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:39,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:39,721 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:39,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:39,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:39,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:39,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:39,724 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:39,724 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:39,724 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:39,724 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:39,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,725 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,726 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:39,726 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:39,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:39,726 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,726 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:39,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:39,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:39,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:39,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:39,727 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:39,727 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:39,727 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:39,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:39,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:39,728 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:39,728 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:39,728 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:39,728 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:39,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:39,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:39,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:39,728 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:39,729 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,729 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,730 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,730 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:39,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c489de4[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:39,732 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:39,732 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,733 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,733 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:39,733 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:39,733 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,733 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:39,733 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:39,735 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,735 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,735 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:39,735 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:39,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:39,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:39,738 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:39,738 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:39,738 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:39,738 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:39,738 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:39,739 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,739 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,739 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,740 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,741 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,743 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,743 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,743 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,743 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:39,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:39,743 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:39,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:39,743 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,743 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:39,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:39,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:39,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,744 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:39,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:39,744 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:39,744 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:39,756 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/0cd17972e9864c7185cac2caf69e08e5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/0cd17972e9864c7185cac2caf69e08e5 2023-04-22 19:56:39,760 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8d85766c3fe2800a849fb8577794261/A of e8d85766c3fe2800a849fb8577794261 into 0cd17972e9864c7185cac2caf69e08e5(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-22 19:56:39,760 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:39,760 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/A, priority=12, startTime=1682193398513; duration=1sec 2023-04-22 19:56:39,760 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:39,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:39,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50664 deadline: 1682193459818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:39,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50704 deadline: 1682193459819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,845 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:39,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:39,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:39,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:39,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:39,848 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:39,848 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:39,848 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:39,849 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:39,849 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,849 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,850 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:39,850 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:39,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:39,850 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,850 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:39,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:39,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:39,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:39,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:39,851 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:39,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:39,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:39,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:39,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:39,852 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:39,852 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:39,852 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:39,852 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:39,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:39,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:39,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:39,852 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:39,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,855 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,855 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:39,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:39,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35667002[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:39,857 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:39,857 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,858 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,858 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,858 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:39,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,858 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:39,858 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:39,858 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:39,859 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:39,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:39,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,859 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:39,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:39,862 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:39,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:39,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:39,862 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:39,862 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:39,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:39,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,863 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:39,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:39,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:39,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:39,864 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,865 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:39,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:39,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,869 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:39,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:39,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:39,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:39,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:39,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:39,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:39,869 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,869 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:39,869 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:39,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:39,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/ebfbb8df23e04211be317705b12106bc 2023-04-22 19:56:39,938 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/786bc68e646942af8e1399d69f4064b5 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/786bc68e646942af8e1399d69f4064b5 2023-04-22 19:56:39,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/786bc68e646942af8e1399d69f4064b5, entries=200, sequenceid=372, filesize=32.3 K 2023-04-22 19:56:39,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/54a75aee345749c08ec4605807d0bdea as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/54a75aee345749c08ec4605807d0bdea 2023-04-22 19:56:39,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/54a75aee345749c08ec4605807d0bdea, entries=150, sequenceid=372, filesize=11.9 K 2023-04-22 19:56:39,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/ebfbb8df23e04211be317705b12106bc as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ebfbb8df23e04211be317705b12106bc 2023-04-22 19:56:39,949 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ebfbb8df23e04211be317705b12106bc, entries=150, sequenceid=372, filesize=11.9 K 2023-04-22 19:56:39,950 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e8d85766c3fe2800a849fb8577794261 in 472ms, sequenceid=372, compaction requested=true 2023-04-22 19:56:39,950 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:39,950 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:39,950 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:56:39,950 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:39,950 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:56:39,950 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:39,951 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85860 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:56:39,951 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:56:39,951 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/A is initiating minor compaction (all files) 2023-04-22 19:56:39,951 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/B is initiating minor compaction (all files) 2023-04-22 19:56:39,951 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/A in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:39,951 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/B in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:39,951 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/0cd17972e9864c7185cac2caf69e08e5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/93f02e12daf94a0d936d3767cdba2405, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/786bc68e646942af8e1399d69f4064b5] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=83.8 K 2023-04-22 19:56:39,952 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/733a27fcda484297a575e0a76d62ba0a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/4baaef097b4c4c6b968454c7e765915d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/54a75aee345749c08ec4605807d0bdea] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=36.4 K 2023-04-22 19:56:39,952 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 0cd17972e9864c7185cac2caf69e08e5, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682193395478 2023-04-22 19:56:39,953 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 93f02e12daf94a0d936d3767cdba2405, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682193397717 2023-04-22 19:56:39,954 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] compactions.Compactor(207): Compacting 786bc68e646942af8e1399d69f4064b5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1682193398859 2023-04-22 19:56:39,954 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 733a27fcda484297a575e0a76d62ba0a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682193395478 2023-04-22 19:56:39,954 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 4baaef097b4c4c6b968454c7e765915d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682193397717 2023-04-22 19:56:39,955 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 54a75aee345749c08ec4605807d0bdea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1682193398859 2023-04-22 19:56:39,962 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#B#compaction#548 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:39,967 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#A#compaction#549 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:39,969 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:39,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:39,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:39,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:39,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:39,980 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:39,980 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:39,980 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:39,980 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:39,980 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,982 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:39,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:39,982 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,982 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:39,982 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:39,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:39,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:39,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:39,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:39,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:39,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:39,989 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:39,989 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:39,989 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:39,992 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:39,992 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:39,992 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:39,992 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:39,992 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 19:56:39,993 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:39,993 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:39,993 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:39,993 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:39,993 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:39,993 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:40,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:40,125 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:40,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:40,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50664 deadline: 1682193460144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:40,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50704 deadline: 1682193460146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:40,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50664 deadline: 1682193460247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:40,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50704 deadline: 1682193460249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,394 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/c6c0cbe371ea421a82eec0d201603bb8 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/c6c0cbe371ea421a82eec0d201603bb8 2023-04-22 19:56:40,401 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8d85766c3fe2800a849fb8577794261/B of e8d85766c3fe2800a849fb8577794261 into c6c0cbe371ea421a82eec0d201603bb8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:40,401 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:40,401 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/B, priority=13, startTime=1682193399950; duration=0sec 2023-04-22 19:56:40,401 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 19:56:40,401 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 19:56:40,402 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 19:56:40,402 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1912): e8d85766c3fe2800a849fb8577794261/C is initiating minor compaction (all files) 2023-04-22 19:56:40,402 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8d85766c3fe2800a849fb8577794261/C in TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:40,402 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/4d5486edca834f739603c2e482c61710, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/063ff417381847a79c65869b9930a1d1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ebfbb8df23e04211be317705b12106bc] into tmpdir=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp, totalSize=36.4 K 2023-04-22 19:56:40,402 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 4d5486edca834f739603c2e482c61710, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682193395478 2023-04-22 19:56:40,403 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting 063ff417381847a79c65869b9930a1d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682193397717 2023-04-22 19:56:40,403 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] compactions.Compactor(207): Compacting ebfbb8df23e04211be317705b12106bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1682193398859 2023-04-22 19:56:40,407 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/ab8dd6309e614a3b9ab11f6c909d9ef6 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/ab8dd6309e614a3b9ab11f6c909d9ef6 2023-04-22 19:56:40,410 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304220acb6d39a327490caba99e0c20138c23 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220acb6d39a327490caba99e0c20138c23 2023-04-22 19:56:40,411 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] throttle.PressureAwareThroughputController(145): e8d85766c3fe2800a849fb8577794261#C#compaction#551 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 19:56:40,415 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8d85766c3fe2800a849fb8577794261/A of e8d85766c3fe2800a849fb8577794261 into ab8dd6309e614a3b9ab11f6c909d9ef6(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:40,415 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:40,415 INFO [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/A, priority=13, startTime=1682193399950; duration=0sec 2023-04-22 19:56:40,415 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:40,421 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/0f22e9a2dd27495ea11eb1e1850c591a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/0f22e9a2dd27495ea11eb1e1850c591a 2023-04-22 19:56:40,424 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8d85766c3fe2800a849fb8577794261/C of e8d85766c3fe2800a849fb8577794261 into 0f22e9a2dd27495ea11eb1e1850c591a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 19:56:40,424 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:40,424 INFO [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261., storeName=e8d85766c3fe2800a849fb8577794261/C, priority=13, startTime=1682193399950; duration=0sec 2023-04-22 19:56:40,424 DEBUG [RS:0;jenkins-hbase4:44791-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:40,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:40,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50664 deadline: 1682193460449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:40,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50704 deadline: 1682193460452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:40,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44791] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50664 deadline: 1682193460752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:40,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50704 deadline: 1682193460756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,818 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=392, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/11eab4ecc8b94b388da949a836f1a484 2023-04-22 19:56:40,828 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/1ec6e211d5354aeda105fe5b7497579f 2023-04-22 19:56:40,838 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/8a17477dd3d74ceb9f9fa01994fc6519 2023-04-22 19:56:40,842 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/11eab4ecc8b94b388da949a836f1a484 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/11eab4ecc8b94b388da949a836f1a484 2023-04-22 19:56:40,845 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/11eab4ecc8b94b388da949a836f1a484, entries=100, sequenceid=392, filesize=18.6 K 2023-04-22 19:56:40,846 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/1ec6e211d5354aeda105fe5b7497579f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/1ec6e211d5354aeda105fe5b7497579f 2023-04-22 19:56:40,849 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/1ec6e211d5354aeda105fe5b7497579f, entries=100, sequenceid=392, filesize=9.5 K 2023-04-22 19:56:40,850 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/8a17477dd3d74ceb9f9fa01994fc6519 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/8a17477dd3d74ceb9f9fa01994fc6519 2023-04-22 19:56:40,853 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/8a17477dd3d74ceb9f9fa01994fc6519, entries=100, sequenceid=392, filesize=9.5 K 2023-04-22 19:56:40,854 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e8d85766c3fe2800a849fb8577794261 in 862ms, sequenceid=392, compaction requested=false 2023-04-22 19:56:40,854 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:40,854 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:40,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:40,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:40,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:40,854 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:40,856 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:40,856 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:40,856 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:40,856 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:40,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:40,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a4b28b8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:40,858 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:40,858 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:40,862 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:40,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:40,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:40,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:40,863 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:40,863 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,863 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:40,865 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:40,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:40,865 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:40,867 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:40,867 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:40,868 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:40,868 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:40,868 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:40,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:40,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:40,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:40,870 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:40,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:40,871 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,873 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:40,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:40,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,877 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:40,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:40,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:40,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:40,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:40,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:40,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:40,877 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:40,877 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:40,877 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:40,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:40,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:40,977 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:40,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:40,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:40,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:40,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:40,981 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:40,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:40,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:40,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:40,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:40,981 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:40,984 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:40,984 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:40,984 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:40,984 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:40,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:40,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:40,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:40,985 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:40,985 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:40,985 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:40,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:40,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:40,986 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 19:56:40,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:40,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:40,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:40,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:40,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:40,986 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:40,998 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304228ab47398f50f41c489d144ecde3f2140 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228ab47398f50f41c489d144ecde3f2140 2023-04-22 19:56:41,003 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=408, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/8205ba4c626c45f28113b0aea6e52a08 2023-04-22 19:56:41,016 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/613d2f970aeb45c88c7376cd7fc1fc93 2023-04-22 19:56:41,029 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/06d1c3e9340b4644ad9b03a40cd44c61 2023-04-22 19:56:41,034 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/8205ba4c626c45f28113b0aea6e52a08 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/8205ba4c626c45f28113b0aea6e52a08 2023-04-22 19:56:41,038 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/8205ba4c626c45f28113b0aea6e52a08, entries=150, sequenceid=408, filesize=25.4 K 2023-04-22 19:56:41,039 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/613d2f970aeb45c88c7376cd7fc1fc93 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/613d2f970aeb45c88c7376cd7fc1fc93 2023-04-22 19:56:41,042 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/613d2f970aeb45c88c7376cd7fc1fc93, entries=150, sequenceid=408, filesize=11.9 K 2023-04-22 19:56:41,042 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/06d1c3e9340b4644ad9b03a40cd44c61 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/06d1c3e9340b4644ad9b03a40cd44c61 2023-04-22 19:56:41,046 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/06d1c3e9340b4644ad9b03a40cd44c61, entries=150, sequenceid=408, filesize=11.9 K 2023-04-22 19:56:41,046 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for e8d85766c3fe2800a849fb8577794261 in 60ms, sequenceid=408, compaction requested=true 2023-04-22 19:56:41,046 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:41,047 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:41,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:41,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:41,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:41,047 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:41,048 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:41,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:41,048 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:41,049 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,049 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:41,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:41,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:41,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:41,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@417d196[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:41,050 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:41,050 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,051 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,051 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:41,051 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:41,051 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:41,052 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,052 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:41,053 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,053 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:41,053 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:41,053 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:41,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:41,055 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:41,055 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:41,055 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:41,055 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:41,055 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:41,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:41,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,058 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:41,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:41,059 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,060 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,062 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:41,062 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:41,062 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:41,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:41,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:41,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:41,062 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:41,063 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:41,062 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:41,063 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:41,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:41,063 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:41,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:41,063 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,063 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,063 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,063 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,163 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:41,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:41,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:41,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:41,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:41,167 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:41,167 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:41,167 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:41,167 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:41,167 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,168 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:41,169 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:41,169 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:41,169 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,169 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:41,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:41,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:41,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:41,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:41,170 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:41,170 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:41,170 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:41,171 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:41,171 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:41,171 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:41,171 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:41,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,172 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:41,173 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,173 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:41,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:41,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cb8db4d[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:41,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:41,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,177 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,177 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:41,177 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:41,177 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:41,177 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,177 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:41,178 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:41,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:41,178 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:41,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:41,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:41,181 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:41,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:41,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:41,181 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:41,181 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:41,183 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,183 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:41,183 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:41,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:41,184 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:41,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,185 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,187 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:41,187 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,187 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:41,187 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:41,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:41,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:41,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:41,187 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:41,187 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:41,188 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,188 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,188 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:41,188 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,188 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:41,188 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:41,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:41,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 19:56:41,288 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:41,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:41,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 19:56:41,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 19:56:41,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 19:56:41,292 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 19:56:41,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:41,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 19:56:41,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 19:56:41,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,292 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:41,294 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:41,294 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:41,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:41,295 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,295 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 19:56:41,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:41,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 19:56:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:41,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 19:56:41,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 19:56:41,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 19:56:41,296 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 19:56:41,296 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 19:56:41,296 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 19:56:41,297 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 19:56:41,297 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 19:56:41,297 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:41,297 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. started... 2023-04-22 19:56:41,297 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-22 19:56:41,298 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:41,298 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:41,298 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:41,298 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:41,298 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:41,298 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:41,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:41,312 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. as already flushing 2023-04-22 19:56:41,314 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304222d1f7723f2b74ab99997b66a45452787 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304222d1f7723f2b74ab99997b66a45452787 2023-04-22 19:56:41,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193401332, max timestamp=1682193401342Num uniques 859; 2023-04-22 19:56:41,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:41,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193401332, max timestamp=1682193401342Num uniques 859; 2023-04-22 19:56:41,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:41,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682193401332, max timestamp=1682193401342Num uniques 859; 2023-04-22 19:56:41,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 19:56:41,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682193401366, max timestamp=1682193401379Num uniques 1018; 2023-04-22 19:56:41,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682193401366, max timestamp=1682193401379Num uniques 1018; 2023-04-22 19:56:41,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 19:56:41,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 19:56:41,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682193401367, max timestamp=1682193401379Num uniques 1018; 2023-04-22 19:56:41,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 19:56:41,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682193401435, max timestamp=1682193401443Num uniques 1077; 2023-04-22 19:56:41,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-22 19:56:41,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682193401435, max timestamp=1682193401443Num uniques 1027; 2023-04-22 19:56:41,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682193401435, max timestamp=1682193401443Num uniques 1077; 2023-04-22 19:56:41,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 19:56:41,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-22 19:56:41,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682193401467, max timestamp=1682193401478Num uniques 1036; 2023-04-22 19:56:41,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-22 19:56:41,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682193401467, max timestamp=1682193401478Num uniques 1036; 2023-04-22 19:56:41,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-22 19:56:41,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682193401467, max timestamp=1682193401478Num uniques 1036; 2023-04-22 19:56:41,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-22 19:56:41,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682193401504, max timestamp=1682193401515Num uniques 1045; 2023-04-22 19:56:41,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-22 19:56:41,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682193401504, max timestamp=1682193401515Num uniques 1045; 2023-04-22 19:56:41,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-22 19:56:41,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682193401504, max timestamp=1682193401515Num uniques 1045; 2023-04-22 19:56:41,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-22 19:56:41,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682193401542, max timestamp=1682193401554Num uniques 1104; 2023-04-22 19:56:41,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682193401542, max timestamp=1682193401554Num uniques 1104; 2023-04-22 19:56:41,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-22 19:56:41,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682193401542, max timestamp=1682193401554Num uniques 1104; 2023-04-22 19:56:41,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-22 19:56:41,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-22 19:56:41,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682193401580, max timestamp=1682193401587Num uniques 1113; 2023-04-22 19:56:41,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-22 19:56:41,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682193401580, max timestamp=1682193401587Num uniques 1113; 2023-04-22 19:56:41,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682193401580, max timestamp=1682193401587Num uniques 1113; 2023-04-22 19:56:41,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-22 19:56:41,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-22 19:56:41,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682193401613, max timestamp=1682193401622Num uniques 1072; 2023-04-22 19:56:41,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-22 19:56:41,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682193401613, max timestamp=1682193401622Num uniques 1072; 2023-04-22 19:56:41,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-22 19:56:41,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682193401613, max timestamp=1682193401622Num uniques 1072; 2023-04-22 19:56:41,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-22 19:56:41,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682193401631, max timestamp=1682193401662Num uniques 1031; 2023-04-22 19:56:41,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-22 19:56:41,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682193401631, max timestamp=1682193401662Num uniques 1031; 2023-04-22 19:56:41,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-22 19:56:41,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682193401631, max timestamp=1682193401662Num uniques 1031; 2023-04-22 19:56:41,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-22 19:56:41,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682193401682, max timestamp=1682193401696Num uniques 1140; 2023-04-22 19:56:41,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-22 19:56:41,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682193401682, max timestamp=1682193401696Num uniques 1140; 2023-04-22 19:56:41,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682193401682, max timestamp=1682193401696Num uniques 1140; 2023-04-22 19:56:41,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-22 19:56:41,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-22 19:56:41,726 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=416, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/00f4268b3ed948f996d567199b63f81d 2023-04-22 19:56:41,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682193401718, max timestamp=1682193401726Num uniques 1099; 2023-04-22 19:56:41,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-22 19:56:41,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682193401718, max timestamp=1682193401726Num uniques 1099; 2023-04-22 19:56:41,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-22 19:56:41,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682193401718, max timestamp=1682193401726Num uniques 1099; 2023-04-22 19:56:41,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-22 19:56:41,738 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/50e01dc354554166b2faa20787541604 2023-04-22 19:56:41,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682193401759, max timestamp=1682193401766Num uniques 1108; 2023-04-22 19:56:41,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-22 19:56:41,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682193401759, max timestamp=1682193401766Num uniques 1108; 2023-04-22 19:56:41,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-22 19:56:41,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682193401759, max timestamp=1682193401766Num uniques 1108; 2023-04-22 19:56:41,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-22 19:56:41,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682193401788, max timestamp=1682193401794Num uniques 1017; 2023-04-22 19:56:41,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-22 19:56:41,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682193401788, max timestamp=1682193401794Num uniques 1017; 2023-04-22 19:56:41,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682193401788, max timestamp=1682193401794Num uniques 1017; 2023-04-22 19:56:41,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-22 19:56:41,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-22 19:56:41,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682193401814, max timestamp=1682193401820Num uniques 1076; 2023-04-22 19:56:41,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682193401814, max timestamp=1682193401820Num uniques 1076; 2023-04-22 19:56:41,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-22 19:56:41,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-22 19:56:41,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682193401814, max timestamp=1682193401820Num uniques 1076; 2023-04-22 19:56:41,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-22 19:56:41,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682193401854, max timestamp=1682193401866Num uniques 1035; 2023-04-22 19:56:41,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-22 19:56:41,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682193401854, max timestamp=1682193401866Num uniques 1035; 2023-04-22 19:56:41,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-22 19:56:41,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682193401854, max timestamp=1682193401866Num uniques 1035; 2023-04-22 19:56:41,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-22 19:56:41,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682193401918, max timestamp=1682193401923Num uniques 1094; 2023-04-22 19:56:41,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-22 19:56:41,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682193401918, max timestamp=1682193401923Num uniques 1094; 2023-04-22 19:56:41,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682193401918, max timestamp=1682193401923Num uniques 1094; 2023-04-22 19:56:41,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-22 19:56:41,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-22 19:56:41,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:41,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44791] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50704 deadline: 1682193461948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 19:56:41,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50646 deadline: 1682193461949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:41,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1682193401940, max timestamp=1682193401951Num uniques 1103; 2023-04-22 19:56:41,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-22 19:56:41,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1682193401940, max timestamp=1682193401951Num uniques 1103; 2023-04-22 19:56:41,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-22 19:56:41,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682193401940, max timestamp=1682193401951Num uniques 1103; 2023-04-22 19:56:41,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-22 19:56:42,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682193401983, max timestamp=1682193402021Num uniques 1062; 2023-04-22 19:56:42,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682193401983, max timestamp=1682193402021Num uniques 1062; 2023-04-22 19:56:42,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-22 19:56:42,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-22 19:56:42,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682193401983, max timestamp=1682193402021Num uniques 1012; 2023-04-22 19:56:42,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-22 19:56:42,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1682193402060, max timestamp=1682193402070Num uniques 1021; 2023-04-22 19:56:42,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-04-22 19:56:42,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1682193402060, max timestamp=1682193402070Num uniques 1021; 2023-04-22 19:56:42,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-04-22 19:56:42,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1682193402060, max timestamp=1682193402070Num uniques 1021; 2023-04-22 19:56:42,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-04-22 19:56:42,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682193402086, max timestamp=1682193402097Num uniques 1080; 2023-04-22 19:56:42,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=8240, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-22 19:56:42,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682193402086, max timestamp=1682193402097Num uniques 1080; 2023-04-22 19:56:42,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682193402086, max timestamp=1682193402097Num uniques 1080; 2023-04-22 19:56:42,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=8240, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-22 19:56:42,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=8240, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-22 19:56:42,105 DEBUG [Thread-3628] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08b57aae to 127.0.0.1:53381 2023-04-22 19:56:42,105 DEBUG [Thread-3628] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:42,105 DEBUG [Thread-3630] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06799455 to 127.0.0.1:53381 2023-04-22 19:56:42,105 DEBUG [Thread-3630] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:42,106 DEBUG [Thread-3621] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1644730f to 127.0.0.1:53381 2023-04-22 19:56:42,106 DEBUG [Thread-3621] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:42,106 DEBUG [Thread-3623] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:53381 2023-04-22 19:56:42,106 DEBUG [Thread-3623] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:42,106 DEBUG [Thread-3634] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:53381 2023-04-22 19:56:42,106 DEBUG [Thread-3634] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:42,109 DEBUG [Thread-3619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45878790 to 127.0.0.1:53381 2023-04-22 19:56:42,109 DEBUG [Thread-3619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:42,109 DEBUG [Thread-3626] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:53381 2023-04-22 19:56:42,109 DEBUG [Thread-3626] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:42,110 DEBUG [Thread-3632] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:53381 2023-04-22 19:56:42,110 DEBUG [Thread-3632] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:42,147 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/57252d4a71b840d38ae83affc493dcbf 2023-04-22 19:56:42,150 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/00f4268b3ed948f996d567199b63f81d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/00f4268b3ed948f996d567199b63f81d 2023-04-22 19:56:42,153 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/00f4268b3ed948f996d567199b63f81d, entries=150, sequenceid=416, filesize=25.4 K 2023-04-22 19:56:42,153 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/50e01dc354554166b2faa20787541604 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/50e01dc354554166b2faa20787541604 2023-04-22 19:56:42,156 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/50e01dc354554166b2faa20787541604, entries=100, sequenceid=416, filesize=9.5 K 2023-04-22 19:56:42,156 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/57252d4a71b840d38ae83affc493dcbf as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/57252d4a71b840d38ae83affc493dcbf 2023-04-22 19:56:42,158 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/57252d4a71b840d38ae83affc493dcbf, entries=100, sequenceid=416, filesize=9.5 K 2023-04-22 19:56:42,159 INFO [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=60.38 KB/61830 for e8d85766c3fe2800a849fb8577794261 in 862ms, sequenceid=416, compaction requested=true 2023-04-22 19:56:42,159 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:42,159 DEBUG [rs(jenkins-hbase4.apache.org,44791,1682193215766)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:42,159 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 19:56:42,159 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 19:56:42,159 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,159 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 19:56:42,159 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 19:56:42,163 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,163 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:42,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:42,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:42,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 19:56:42,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:42,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:42,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:42,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44791,1682193215766' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 19:56:42,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ad84cb8[Count = 0] remaining members to acquire global barrier 2023-04-22 19:56:42,165 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 19:56:42,165 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:42,167 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:42,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:42,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 19:56:42,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 19:56:42,167 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,167 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 19:56:42,167 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44791,1682193215766' in zk 2023-04-22 19:56:42,169 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 19:56:42,169 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,169 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:42,169 DEBUG [member: 'jenkins-hbase4.apache.org,44791,1682193215766' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 19:56:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44791,1682193215766': 2023-04-22 19:56:42,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44791,1682193215766' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 19:56:42,170 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 19:56:42,171 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 19:56:42,171 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 19:56:42,171 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 19:56:42,171 INFO [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 19:56:42,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:42,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 19:56:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 19:56:42,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:42,172 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 19:56:42,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:42,172 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 19:56:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:42,173 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 19:56:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 19:56:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 19:56:42,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 19:56:42,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:42,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 19:56:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 19:56:42,175 DEBUG [(jenkins-hbase4.apache.org,34929,1682193214523)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 19:56:42,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 19:56:42,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 19:56:42,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 19:56:42,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 19:56:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 19:56:42,175 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:42,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:42,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 19:56:42,176 DEBUG [Thread-3625] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 19:56:42,176 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 19:56:42,176 DEBUG [Thread-3625] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 19:56:42,276 DEBUG [Thread-3625] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 19:56:42,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 19:56:46,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44791] regionserver.HRegion(9158): Flush requested on e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:46,614 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=67.09 KB heapSize=183.80 KB 2023-04-22 19:56:46,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:46,614 DEBUG [Thread-3617] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68067de1 to 127.0.0.1:53381 2023-04-22 19:56:46,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:46,614 DEBUG [Thread-3617] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:46,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:46,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:46,615 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:46,615 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 19:56:46,617 DEBUG [Thread-3615] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e8824d7 to 127.0.0.1:53381 2023-04-22 19:56:46,617 DEBUG [Thread-3615] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 215 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 238 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(394): wrote 147 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 4396 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 4434 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 4472 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 4420 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(398): read 4416 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 19:56:46,618 INFO [Listener at localhost/46665] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 19:56:46,618 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3011deb3 to 127.0.0.1:53381 2023-04-22 19:56:46,618 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:46,619 INFO [Listener at localhost/46665] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 19:56:46,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 19:56:46,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:46,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 19:56:46,622 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193406622"}]},"ts":"1682193406622"} 2023-04-22 19:56:46,623 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 19:56:46,626 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 19:56:46,626 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8d85766c3fe2800a849fb8577794261, UNASSIGN}] 2023-04-22 19:56:46,627 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8d85766c3fe2800a849fb8577794261, UNASSIGN 2023-04-22 19:56:46,628 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=e8d85766c3fe2800a849fb8577794261, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:46,628 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193406628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682193406628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682193406628"}]},"ts":"1682193406628"} 2023-04-22 19:56:46,629 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766}] 2023-04-22 19:56:46,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 19:56:46,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:46,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e8d85766c3fe2800a849fb8577794261, disabling compactions & flushes 2023-04-22 19:56:46,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:46,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 19:56:47,027 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422d5af4e143b7048fabe6b392fc46117cb to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d5af4e143b7048fabe6b392fc46117cb 2023-04-22 19:56:47,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 19:56:47,432 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=769, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/35b71d1256bc44b299cd45a5859578ab 2023-04-22 19:56:47,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 19:56:47,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=769 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/ae96d9543e9e47cc81bc33bb5d2bb3bc 2023-04-22 19:56:48,248 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=769 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/f63d2876eb3f465f9e9cce258f3175d1 2023-04-22 19:56:48,252 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/35b71d1256bc44b299cd45a5859578ab as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/35b71d1256bc44b299cd45a5859578ab 2023-04-22 19:56:48,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/35b71d1256bc44b299cd45a5859578ab, entries=150, sequenceid=769, filesize=25.4 K 2023-04-22 19:56:48,255 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/ae96d9543e9e47cc81bc33bb5d2bb3bc as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ae96d9543e9e47cc81bc33bb5d2bb3bc 2023-04-22 19:56:48,258 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ae96d9543e9e47cc81bc33bb5d2bb3bc, entries=150, sequenceid=769, filesize=11.9 K 2023-04-22 19:56:48,258 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/f63d2876eb3f465f9e9cce258f3175d1 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f63d2876eb3f465f9e9cce258f3175d1 2023-04-22 19:56:48,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f63d2876eb3f465f9e9cce258f3175d1, entries=150, sequenceid=769, filesize=11.9 K 2023-04-22 19:56:48,262 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~183.75 KB/188160, currentSize=6.71 KB/6870 for e8d85766c3fe2800a849fb8577794261 in 1648ms, sequenceid=769, compaction requested=true 2023-04-22 19:56:48,262 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:48,262 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:48,262 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. because compaction request was cancelled 2023-04-22 19:56:48,262 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:48,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:48,262 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. after waiting 0 ms 2023-04-22 19:56:48,262 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. because compaction request was cancelled 2023-04-22 19:56:48,262 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:48,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:48,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 19:56:48,262 DEBUG [RS:0;jenkins-hbase4:44791-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. because compaction request was cancelled 2023-04-22 19:56:48,262 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e8d85766c3fe2800a849fb8577794261 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-22 19:56:48,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=A 2023-04-22 19:56:48,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:48,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=B 2023-04-22 19:56:48,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:48,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8d85766c3fe2800a849fb8577794261, store=C 2023-04-22 19:56:48,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 19:56:48,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304225b505309144a4fae8b01d7a663e78ac8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225b505309144a4fae8b01d7a663e78ac8 2023-04-22 19:56:48,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 19:56:49,077 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=773, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/48cc6d986e34417a84c28d7f9f8df01b 2023-04-22 19:56:49,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=773 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/ded1f0664349490cb5313bfb531b4b7f 2023-04-22 19:56:49,893 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=773 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/35f3b41e3a1743af93543a7a75f588aa 2023-04-22 19:56:49,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/A/48cc6d986e34417a84c28d7f9f8df01b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/48cc6d986e34417a84c28d7f9f8df01b 2023-04-22 19:56:49,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/48cc6d986e34417a84c28d7f9f8df01b, entries=50, sequenceid=773, filesize=11.7 K 2023-04-22 19:56:49,900 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/B/ded1f0664349490cb5313bfb531b4b7f as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ded1f0664349490cb5313bfb531b4b7f 2023-04-22 19:56:49,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ded1f0664349490cb5313bfb531b4b7f, entries=50, sequenceid=773, filesize=7.1 K 2023-04-22 19:56:49,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/.tmp/C/35f3b41e3a1743af93543a7a75f588aa as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/35f3b41e3a1743af93543a7a75f588aa 2023-04-22 19:56:49,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/35f3b41e3a1743af93543a7a75f588aa, entries=50, sequenceid=773, filesize=7.1 K 2023-04-22 19:56:49,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=0 B/0 for e8d85766c3fe2800a849fb8577794261 in 1661ms, sequenceid=773, compaction requested=true 2023-04-22 19:56:49,924 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/71a9ee20a6764393b15f4885cd349c1c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/46a8ab2dc6084416967460f17aab9ded, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/6fe8d27774984c058e4478fa20ce8f22, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/fbdf1ff6f00541178ec185d35e6a12f5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/438b6a3f24ef47debf744856e6590c84, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/145038b32a9d488fb87cf0a27cf63393, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c81e20131e1f48aa972ffa459f67fc8c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/a984a31c437740769b39a6e4d6611017, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/45be71951bde41118df6a1182c18fb66, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/63dc634ff8794a288cb4f157bc667745, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/088baf6b8a1c451691b3b0f9c7af30ec, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/0907a6bb931c4595acd165baa3651e56, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/24f9b879298d4d4ebce63c2d7d1b509f, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/b6f2588587b143a3a1cd37553018437c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/9d3c8766a80e4c86a5e04f8585716025, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c0de3ef26346424fb879d916b3597aed, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/3112c90c460540b68ce3c5535839f18c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/e1f29bd0de8a4054889fb258a4f2f670, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/ba3e4c632c534426b7686c878182d292, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/18c56a4b77d540aa93a40dde5ab28105, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/0cd17972e9864c7185cac2caf69e08e5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/93f02e12daf94a0d936d3767cdba2405, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/786bc68e646942af8e1399d69f4064b5] to archive 2023-04-22 19:56:49,924 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:56:49,926 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/71a9ee20a6764393b15f4885cd349c1c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/71a9ee20a6764393b15f4885cd349c1c 2023-04-22 19:56:49,927 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/46a8ab2dc6084416967460f17aab9ded to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/46a8ab2dc6084416967460f17aab9ded 2023-04-22 19:56:49,928 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/6fe8d27774984c058e4478fa20ce8f22 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/6fe8d27774984c058e4478fa20ce8f22 2023-04-22 19:56:49,928 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/fbdf1ff6f00541178ec185d35e6a12f5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/fbdf1ff6f00541178ec185d35e6a12f5 2023-04-22 19:56:49,929 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/438b6a3f24ef47debf744856e6590c84 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/438b6a3f24ef47debf744856e6590c84 2023-04-22 19:56:49,930 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/145038b32a9d488fb87cf0a27cf63393 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/145038b32a9d488fb87cf0a27cf63393 2023-04-22 19:56:49,931 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c81e20131e1f48aa972ffa459f67fc8c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c81e20131e1f48aa972ffa459f67fc8c 2023-04-22 19:56:49,932 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/a984a31c437740769b39a6e4d6611017 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/a984a31c437740769b39a6e4d6611017 2023-04-22 19:56:49,933 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/45be71951bde41118df6a1182c18fb66 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/45be71951bde41118df6a1182c18fb66 2023-04-22 19:56:49,934 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/63dc634ff8794a288cb4f157bc667745 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/63dc634ff8794a288cb4f157bc667745 2023-04-22 19:56:49,934 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/088baf6b8a1c451691b3b0f9c7af30ec to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/088baf6b8a1c451691b3b0f9c7af30ec 2023-04-22 19:56:49,935 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/0907a6bb931c4595acd165baa3651e56 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/0907a6bb931c4595acd165baa3651e56 2023-04-22 19:56:49,936 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/24f9b879298d4d4ebce63c2d7d1b509f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/24f9b879298d4d4ebce63c2d7d1b509f 2023-04-22 19:56:49,937 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/b6f2588587b143a3a1cd37553018437c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/b6f2588587b143a3a1cd37553018437c 2023-04-22 19:56:49,938 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/9d3c8766a80e4c86a5e04f8585716025 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/9d3c8766a80e4c86a5e04f8585716025 2023-04-22 19:56:49,938 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c0de3ef26346424fb879d916b3597aed to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/c0de3ef26346424fb879d916b3597aed 2023-04-22 19:56:49,939 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/3112c90c460540b68ce3c5535839f18c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/3112c90c460540b68ce3c5535839f18c 2023-04-22 19:56:49,940 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/e1f29bd0de8a4054889fb258a4f2f670 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/e1f29bd0de8a4054889fb258a4f2f670 2023-04-22 19:56:49,941 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/ba3e4c632c534426b7686c878182d292 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/ba3e4c632c534426b7686c878182d292 2023-04-22 19:56:49,942 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/18c56a4b77d540aa93a40dde5ab28105 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/18c56a4b77d540aa93a40dde5ab28105 2023-04-22 19:56:49,943 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/0cd17972e9864c7185cac2caf69e08e5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/0cd17972e9864c7185cac2caf69e08e5 2023-04-22 19:56:49,943 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/93f02e12daf94a0d936d3767cdba2405 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/93f02e12daf94a0d936d3767cdba2405 2023-04-22 19:56:49,944 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/786bc68e646942af8e1399d69f4064b5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/786bc68e646942af8e1399d69f4064b5 2023-04-22 19:56:49,947 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/bc1eeebd41584c18aa13d6862defa13a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/6fc8ef7385fe4782a8cf9ea2f46eec1d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/039044d3bdab43fbad17f0c43d5ef85b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0109ddb8f20e4b7bac204baad88662c4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0f1b26369f7d4fcc8eb220e1f62cc7b1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/92d0a54043b94838b75ce172f58c3d99, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/a689097420c54720add9e01c901a0685, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ada350b365784baa93a85ef40ba50f5c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/f1ff711ed6914a829afff5541ed92c13, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/3bb364ccb15f41de96c002b4cfbf3948, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/001f8a6c3e9a40b589597e645866a870, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0c420fb0d2e848c487a4b06342865ee5, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e2234b0ad21f4d068a7a6f66817c1b8b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/cfa31a96ce7d43ba8bc14e72d76c45c9, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e21b43fdfe964e81b811189d7dec10d2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/1a00f9e089704d2ca006c1a72b6ef871, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/58c79b7d406c46d38128bd40935b2d30, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/68a4b2a8f9ec41889648588d5a5bc460, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/a27818f710b740ff9cb85be9512992c2, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/733a27fcda484297a575e0a76d62ba0a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/056e8f8f490a456084ecbd3b2e9da10b, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/4baaef097b4c4c6b968454c7e765915d, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/54a75aee345749c08ec4605807d0bdea] to archive 2023-04-22 19:56:49,947 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:56:49,949 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/bc1eeebd41584c18aa13d6862defa13a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/bc1eeebd41584c18aa13d6862defa13a 2023-04-22 19:56:49,950 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/6fc8ef7385fe4782a8cf9ea2f46eec1d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/6fc8ef7385fe4782a8cf9ea2f46eec1d 2023-04-22 19:56:49,951 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/039044d3bdab43fbad17f0c43d5ef85b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/039044d3bdab43fbad17f0c43d5ef85b 2023-04-22 19:56:49,952 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0109ddb8f20e4b7bac204baad88662c4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0109ddb8f20e4b7bac204baad88662c4 2023-04-22 19:56:49,953 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0f1b26369f7d4fcc8eb220e1f62cc7b1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0f1b26369f7d4fcc8eb220e1f62cc7b1 2023-04-22 19:56:49,954 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/92d0a54043b94838b75ce172f58c3d99 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/92d0a54043b94838b75ce172f58c3d99 2023-04-22 19:56:49,955 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/a689097420c54720add9e01c901a0685 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/a689097420c54720add9e01c901a0685 2023-04-22 19:56:49,956 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ada350b365784baa93a85ef40ba50f5c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ada350b365784baa93a85ef40ba50f5c 2023-04-22 19:56:49,957 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/f1ff711ed6914a829afff5541ed92c13 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/f1ff711ed6914a829afff5541ed92c13 2023-04-22 19:56:49,958 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/3bb364ccb15f41de96c002b4cfbf3948 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/3bb364ccb15f41de96c002b4cfbf3948 2023-04-22 19:56:49,959 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/001f8a6c3e9a40b589597e645866a870 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/001f8a6c3e9a40b589597e645866a870 2023-04-22 19:56:49,960 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0c420fb0d2e848c487a4b06342865ee5 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/0c420fb0d2e848c487a4b06342865ee5 2023-04-22 19:56:49,961 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e2234b0ad21f4d068a7a6f66817c1b8b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e2234b0ad21f4d068a7a6f66817c1b8b 2023-04-22 19:56:49,962 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/cfa31a96ce7d43ba8bc14e72d76c45c9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/cfa31a96ce7d43ba8bc14e72d76c45c9 2023-04-22 19:56:49,963 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e21b43fdfe964e81b811189d7dec10d2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/e21b43fdfe964e81b811189d7dec10d2 2023-04-22 19:56:49,964 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/1a00f9e089704d2ca006c1a72b6ef871 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/1a00f9e089704d2ca006c1a72b6ef871 2023-04-22 19:56:49,964 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/58c79b7d406c46d38128bd40935b2d30 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/58c79b7d406c46d38128bd40935b2d30 2023-04-22 19:56:49,965 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/68a4b2a8f9ec41889648588d5a5bc460 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/68a4b2a8f9ec41889648588d5a5bc460 2023-04-22 19:56:49,966 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/a27818f710b740ff9cb85be9512992c2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/a27818f710b740ff9cb85be9512992c2 2023-04-22 19:56:49,967 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/733a27fcda484297a575e0a76d62ba0a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/733a27fcda484297a575e0a76d62ba0a 2023-04-22 19:56:49,968 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/056e8f8f490a456084ecbd3b2e9da10b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/056e8f8f490a456084ecbd3b2e9da10b 2023-04-22 19:56:49,969 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/4baaef097b4c4c6b968454c7e765915d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/4baaef097b4c4c6b968454c7e765915d 2023-04-22 19:56:49,970 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/54a75aee345749c08ec4605807d0bdea to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/54a75aee345749c08ec4605807d0bdea 2023-04-22 19:56:49,973 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/dfc581b1160b484ca38c2f041eb65387, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b2c46874a9004108bd16d8ebeb1caa23, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/59d71275b4974794a82ba6cafa9e590c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/c0b1314510304a01b4a11f388a4eca1e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/928d11612f0c44f7ad7396df5688e32c, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b140cffa802a405e95f61f8e20595f2e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/7bc4b56bad464616b4a92575f8be33f4, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ce932dce1df54501920060684c5ebd06, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e6f1e7a5a70246a19df914a682e5ff7a, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/070dfbbd4afb4a21815004a6de27f1a7, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/01d7bee271e84e7ea4b03e5baeef8dce, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/912a63f39d0f4bdc8cb5fa3f28053384, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/a076067d9fae4b5db3fc617c71fbaa75, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ff1ab19d535c4356b214fcb89cd6b6d1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/02fd166ad91c4c21aac1d18450e9ed70, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f9275824e4f6480c9ab353a190a8ac50, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/095b8929cc4b4fa19d1ea6d3d4d0386e, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/2837ed608a014ae7978801e17b099664, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e1bb3c0a5954441f83c79ff499af70cc, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/4d5486edca834f739603c2e482c61710, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f1b7d853d15b4e2fa88886c18674d466, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/063ff417381847a79c65869b9930a1d1, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ebfbb8df23e04211be317705b12106bc] to archive 2023-04-22 19:56:49,974 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 19:56:49,975 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/dfc581b1160b484ca38c2f041eb65387 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/dfc581b1160b484ca38c2f041eb65387 2023-04-22 19:56:49,976 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b2c46874a9004108bd16d8ebeb1caa23 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b2c46874a9004108bd16d8ebeb1caa23 2023-04-22 19:56:49,977 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/59d71275b4974794a82ba6cafa9e590c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/59d71275b4974794a82ba6cafa9e590c 2023-04-22 19:56:49,978 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/c0b1314510304a01b4a11f388a4eca1e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/c0b1314510304a01b4a11f388a4eca1e 2023-04-22 19:56:49,979 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/928d11612f0c44f7ad7396df5688e32c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/928d11612f0c44f7ad7396df5688e32c 2023-04-22 19:56:49,980 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b140cffa802a405e95f61f8e20595f2e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/b140cffa802a405e95f61f8e20595f2e 2023-04-22 19:56:49,981 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/7bc4b56bad464616b4a92575f8be33f4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/7bc4b56bad464616b4a92575f8be33f4 2023-04-22 19:56:49,982 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ce932dce1df54501920060684c5ebd06 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ce932dce1df54501920060684c5ebd06 2023-04-22 19:56:49,983 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e6f1e7a5a70246a19df914a682e5ff7a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e6f1e7a5a70246a19df914a682e5ff7a 2023-04-22 19:56:49,984 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/070dfbbd4afb4a21815004a6de27f1a7 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/070dfbbd4afb4a21815004a6de27f1a7 2023-04-22 19:56:49,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/01d7bee271e84e7ea4b03e5baeef8dce to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/01d7bee271e84e7ea4b03e5baeef8dce 2023-04-22 19:56:49,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/912a63f39d0f4bdc8cb5fa3f28053384 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/912a63f39d0f4bdc8cb5fa3f28053384 2023-04-22 19:56:49,986 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/a076067d9fae4b5db3fc617c71fbaa75 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/a076067d9fae4b5db3fc617c71fbaa75 2023-04-22 19:56:49,987 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ff1ab19d535c4356b214fcb89cd6b6d1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ff1ab19d535c4356b214fcb89cd6b6d1 2023-04-22 19:56:49,988 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/02fd166ad91c4c21aac1d18450e9ed70 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/02fd166ad91c4c21aac1d18450e9ed70 2023-04-22 19:56:49,989 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f9275824e4f6480c9ab353a190a8ac50 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f9275824e4f6480c9ab353a190a8ac50 2023-04-22 19:56:49,990 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/095b8929cc4b4fa19d1ea6d3d4d0386e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/095b8929cc4b4fa19d1ea6d3d4d0386e 2023-04-22 19:56:49,991 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/2837ed608a014ae7978801e17b099664 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/2837ed608a014ae7978801e17b099664 2023-04-22 19:56:49,992 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e1bb3c0a5954441f83c79ff499af70cc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/e1bb3c0a5954441f83c79ff499af70cc 2023-04-22 19:56:49,993 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/4d5486edca834f739603c2e482c61710 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/4d5486edca834f739603c2e482c61710 2023-04-22 19:56:49,994 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f1b7d853d15b4e2fa88886c18674d466 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f1b7d853d15b4e2fa88886c18674d466 2023-04-22 19:56:49,995 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/063ff417381847a79c65869b9930a1d1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/063ff417381847a79c65869b9930a1d1 2023-04-22 19:56:49,995 DEBUG [StoreCloser-TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ebfbb8df23e04211be317705b12106bc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/ebfbb8df23e04211be317705b12106bc 2023-04-22 19:56:49,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/recovered.edits/776.seqid, newMaxSeqId=776, maxSeqId=4 2023-04-22 19:56:50,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261. 2023-04-22 19:56:50,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e8d85766c3fe2800a849fb8577794261: 2023-04-22 19:56:50,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:50,002 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=e8d85766c3fe2800a849fb8577794261, regionState=CLOSED 2023-04-22 19:56:50,002 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682193410002"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682193410002"}]},"ts":"1682193410002"} 2023-04-22 19:56:50,004 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-22 19:56:50,004 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure e8d85766c3fe2800a849fb8577794261, server=jenkins-hbase4.apache.org,44791,1682193215766 in 3.3740 sec 2023-04-22 19:56:50,005 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-22 19:56:50,005 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8d85766c3fe2800a849fb8577794261, UNASSIGN in 3.3780 sec 2023-04-22 19:56:50,005 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682193410005"}]},"ts":"1682193410005"} 2023-04-22 19:56:50,006 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 19:56:50,007 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 19:56:50,009 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.3890 sec 2023-04-22 19:56:50,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 19:56:50,728 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-22 19:56:50,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 19:56:50,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:50,731 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:50,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-22 19:56:50,731 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:50,736 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:50,738 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C, FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/recovered.edits] 2023-04-22 19:56:50,740 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/00f4268b3ed948f996d567199b63f81d to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/00f4268b3ed948f996d567199b63f81d 2023-04-22 19:56:50,741 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/11eab4ecc8b94b388da949a836f1a484 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/11eab4ecc8b94b388da949a836f1a484 2023-04-22 19:56:50,742 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/35b71d1256bc44b299cd45a5859578ab to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/35b71d1256bc44b299cd45a5859578ab 2023-04-22 19:56:50,743 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/48cc6d986e34417a84c28d7f9f8df01b to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/48cc6d986e34417a84c28d7f9f8df01b 2023-04-22 19:56:50,744 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/8205ba4c626c45f28113b0aea6e52a08 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/8205ba4c626c45f28113b0aea6e52a08 2023-04-22 19:56:50,745 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/ab8dd6309e614a3b9ab11f6c909d9ef6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/A/ab8dd6309e614a3b9ab11f6c909d9ef6 2023-04-22 19:56:50,747 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/1ec6e211d5354aeda105fe5b7497579f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/1ec6e211d5354aeda105fe5b7497579f 2023-04-22 19:56:50,748 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/50e01dc354554166b2faa20787541604 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/50e01dc354554166b2faa20787541604 2023-04-22 19:56:50,749 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/613d2f970aeb45c88c7376cd7fc1fc93 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/613d2f970aeb45c88c7376cd7fc1fc93 2023-04-22 19:56:50,750 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ae96d9543e9e47cc81bc33bb5d2bb3bc to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ae96d9543e9e47cc81bc33bb5d2bb3bc 2023-04-22 19:56:50,751 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/c6c0cbe371ea421a82eec0d201603bb8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/c6c0cbe371ea421a82eec0d201603bb8 2023-04-22 19:56:50,752 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ded1f0664349490cb5313bfb531b4b7f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/B/ded1f0664349490cb5313bfb531b4b7f 2023-04-22 19:56:50,754 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/06d1c3e9340b4644ad9b03a40cd44c61 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/06d1c3e9340b4644ad9b03a40cd44c61 2023-04-22 19:56:50,755 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/0f22e9a2dd27495ea11eb1e1850c591a to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/0f22e9a2dd27495ea11eb1e1850c591a 2023-04-22 19:56:50,755 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/35f3b41e3a1743af93543a7a75f588aa to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/35f3b41e3a1743af93543a7a75f588aa 2023-04-22 19:56:50,757 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/57252d4a71b840d38ae83affc493dcbf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/57252d4a71b840d38ae83affc493dcbf 2023-04-22 19:56:50,758 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/8a17477dd3d74ceb9f9fa01994fc6519 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/8a17477dd3d74ceb9f9fa01994fc6519 2023-04-22 19:56:50,759 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f63d2876eb3f465f9e9cce258f3175d1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/C/f63d2876eb3f465f9e9cce258f3175d1 2023-04-22 19:56:50,761 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/recovered.edits/776.seqid to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261/recovered.edits/776.seqid 2023-04-22 19:56:50,761 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/.tmp/data/default/TestAcidGuarantees/e8d85766c3fe2800a849fb8577794261 2023-04-22 19:56:50,761 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 19:56:50,762 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 19:56:50,762 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-22 19:56:50,765 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220404a5968688453b9eff4223a27f344f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220404a5968688453b9eff4223a27f344f 2023-04-22 19:56:50,766 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220acb6d39a327490caba99e0c20138c23 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220acb6d39a327490caba99e0c20138c23 2023-04-22 19:56:50,766 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220ffc5ec523264eaa949803247edaf8e1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220ffc5ec523264eaa949803247edaf8e1 2023-04-22 19:56:50,767 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304222828521e85824534a3e982baf59551b9 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304222828521e85824534a3e982baf59551b9 2023-04-22 19:56:50,768 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042229db8b866c404fab9d445091fa973ec1 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042229db8b866c404fab9d445091fa973ec1 2023-04-22 19:56:50,769 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304222d1f7723f2b74ab99997b66a45452787 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304222d1f7723f2b74ab99997b66a45452787 2023-04-22 19:56:50,770 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223451950209084863aeb8ea675cef67cf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223451950209084863aeb8ea675cef67cf 2023-04-22 19:56:50,771 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223ebba83edba54af4a814b3524bf94f3f to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223ebba83edba54af4a814b3524bf94f3f 2023-04-22 19:56:50,772 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225b505309144a4fae8b01d7a663e78ac8 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225b505309144a4fae8b01d7a663e78ac8 2023-04-22 19:56:50,773 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042264a77d8f65ad42729bb9d24755edc661 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042264a77d8f65ad42729bb9d24755edc661 2023-04-22 19:56:50,774 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228ab47398f50f41c489d144ecde3f2140 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228ab47398f50f41c489d144ecde3f2140 2023-04-22 19:56:50,775 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a40238fcc11347298b41a9aa30c64a55 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a40238fcc11347298b41a9aa30c64a55 2023-04-22 19:56:50,776 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b9af0cc985a942718de1a892aca8a977 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b9af0cc985a942718de1a892aca8a977 2023-04-22 19:56:50,777 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c1b68ad177944f36984cf28e1ce7e99e to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c1b68ad177944f36984cf28e1ce7e99e 2023-04-22 19:56:50,778 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c215f197ebfd46c7bdc00ead024866ed to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c215f197ebfd46c7bdc00ead024866ed 2023-04-22 19:56:50,779 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c9b75bf2bbf14622aee57f268a83ea51 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c9b75bf2bbf14622aee57f268a83ea51 2023-04-22 19:56:50,780 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cd00dd7e50d2495586c20cc8585ec2b4 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cd00dd7e50d2495586c20cc8585ec2b4 2023-04-22 19:56:50,781 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cfb7483d1a88493581c1aba8c8b715cf to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422cfb7483d1a88493581c1aba8c8b715cf 2023-04-22 19:56:50,782 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d5af4e143b7048fabe6b392fc46117cb to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d5af4e143b7048fabe6b392fc46117cb 2023-04-22 19:56:50,783 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d9d3b0b9d355447ba5e59fe11105cbe6 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d9d3b0b9d355447ba5e59fe11105cbe6 2023-04-22 19:56:50,784 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f01e9365d78a406092a460a3024cc54c to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f01e9365d78a406092a460a3024cc54c 2023-04-22 19:56:50,785 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f63448a994e340aa944c7117866556ea to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f63448a994e340aa944c7117866556ea 2023-04-22 19:56:50,785 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422fa947b0ae77640b287a1d5c1804dcde2 to hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422fa947b0ae77640b287a1d5c1804dcde2 2023-04-22 19:56:50,786 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 19:56:50,788 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:50,789 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 19:56:50,790 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 19:56:50,791 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:50,791 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 19:56:50,791 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682193410791"}]},"ts":"9223372036854775807"} 2023-04-22 19:56:50,792 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 19:56:50,793 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e8d85766c3fe2800a849fb8577794261, NAME => 'TestAcidGuarantees,,1682193379009.e8d85766c3fe2800a849fb8577794261.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 19:56:50,793 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 19:56:50,793 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682193410793"}]},"ts":"9223372036854775807"} 2023-04-22 19:56:50,794 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 19:56:50,795 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 19:56:50,796 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 66 msec 2023-04-22 19:56:50,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34929] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-22 19:56:50,832 INFO [Listener at localhost/46665] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-22 19:56:50,843 INFO [Listener at localhost/46665] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=226 (was 225) - Thread LEAK? -, OpenFileDescriptor=573 (was 524) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=420 (was 397) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12797 (was 12849) 2023-04-22 19:56:50,844 INFO [Listener at localhost/46665] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-22 19:56:50,844 INFO [Listener at localhost/46665] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 19:56:50,844 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69cada93 to 127.0.0.1:53381 2023-04-22 19:56:50,844 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:50,845 DEBUG [Listener at localhost/46665] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-22 19:56:50,845 DEBUG [Listener at localhost/46665] util.JVMClusterUtil(257): Found active master hash=1391574506, stopped=false 2023-04-22 19:56:50,845 INFO [Listener at localhost/46665] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,34929,1682193214523 2023-04-22 19:56:50,846 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-22 19:56:50,846 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-22 19:56:50,846 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 19:56:50,846 INFO [Listener at localhost/46665] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 19:56:50,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 19:56:50,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 19:56:50,847 DEBUG [Listener at localhost/46665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7b928b0c to 127.0.0.1:53381 2023-04-22 19:56:50,847 DEBUG [Listener at localhost/46665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:50,847 INFO [Listener at localhost/46665] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44791,1682193215766' ***** 2023-04-22 19:56:50,847 INFO [Listener at localhost/46665] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-22 19:56:50,847 INFO [RS:0;jenkins-hbase4:44791] regionserver.HeapMemoryManager(220): Stopping 2023-04-22 19:56:50,847 INFO [RS:0;jenkins-hbase4:44791] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-22 19:56:50,848 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-22 19:56:50,848 INFO [RS:0;jenkins-hbase4:44791] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-22 19:56:50,848 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-22 19:56:50,848 INFO [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(3304): Received CLOSE for 163905e55735ce7abf13fc939c703557 2023-04-22 19:56:50,848 INFO [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:50,848 DEBUG [RS:0;jenkins-hbase4:44791] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x141d15b9 to 127.0.0.1:53381 2023-04-22 19:56:50,848 DEBUG [RS:0;jenkins-hbase4:44791] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:50,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 163905e55735ce7abf13fc939c703557, disabling compactions & flushes 2023-04-22 19:56:50,849 INFO [RS:0;jenkins-hbase4:44791] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-22 19:56:50,849 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:56:50,849 INFO [RS:0;jenkins-hbase4:44791] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-22 19:56:50,849 INFO [RS:0;jenkins-hbase4:44791] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-22 19:56:50,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:56:50,849 INFO [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-22 19:56:50,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. after waiting 0 ms 2023-04-22 19:56:50,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:56:50,849 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 163905e55735ce7abf13fc939c703557 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-22 19:56:50,849 INFO [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-22 19:56:50,849 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1479): Online Regions={163905e55735ce7abf13fc939c703557=hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557., 1588230740=hbase:meta,,1.1588230740} 2023-04-22 19:56:50,850 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-22 19:56:50,850 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-22 19:56:50,850 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-22 19:56:50,850 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-22 19:56:50,850 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-22 19:56:50,850 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-22 19:56:50,852 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1505): Waiting on 1588230740, 163905e55735ce7abf13fc939c703557 2023-04-22 19:56:50,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/namespace/163905e55735ce7abf13fc939c703557/.tmp/info/b332a43c463441a3a58b8f73ab03724a 2023-04-22 19:56:50,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/namespace/163905e55735ce7abf13fc939c703557/.tmp/info/b332a43c463441a3a58b8f73ab03724a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/namespace/163905e55735ce7abf13fc939c703557/info/b332a43c463441a3a58b8f73ab03724a 2023-04-22 19:56:50,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/namespace/163905e55735ce7abf13fc939c703557/info/b332a43c463441a3a58b8f73ab03724a, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 19:56:50,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 163905e55735ce7abf13fc939c703557 in 21ms, sequenceid=6, compaction requested=false 2023-04-22 19:56:50,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/namespace/163905e55735ce7abf13fc939c703557/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-22 19:56:50,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:56:50,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 163905e55735ce7abf13fc939c703557: 2023-04-22 19:56:50,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682193218866.163905e55735ce7abf13fc939c703557. 2023-04-22 19:56:51,052 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-22 19:56:51,252 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-22 19:56:51,262 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/.tmp/info/e7faa5bcb8dd494aa86be35df5eaeba9 2023-04-22 19:56:51,266 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e7faa5bcb8dd494aa86be35df5eaeba9 2023-04-22 19:56:51,453 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-22 19:56:51,501 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-22 19:56:51,501 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-22 19:56:51,653 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-22 19:56:51,671 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/.tmp/rep_barrier/8f3d96068af94914b56452243b07572a 2023-04-22 19:56:51,674 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8f3d96068af94914b56452243b07572a 2023-04-22 19:56:51,853 INFO [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-22 19:56:51,853 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-22 19:56:51,854 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-22 19:56:52,054 DEBUG [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-22 19:56:52,080 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/.tmp/table/7fd4551af47e413499f7c6db870ad10d 2023-04-22 19:56:52,083 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7fd4551af47e413499f7c6db870ad10d 2023-04-22 19:56:52,083 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/.tmp/info/e7faa5bcb8dd494aa86be35df5eaeba9 as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/info/e7faa5bcb8dd494aa86be35df5eaeba9 2023-04-22 19:56:52,086 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e7faa5bcb8dd494aa86be35df5eaeba9 2023-04-22 19:56:52,086 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/info/e7faa5bcb8dd494aa86be35df5eaeba9, entries=22, sequenceid=93, filesize=7.1 K 2023-04-22 19:56:52,087 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/.tmp/rep_barrier/8f3d96068af94914b56452243b07572a as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/rep_barrier/8f3d96068af94914b56452243b07572a 2023-04-22 19:56:52,090 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8f3d96068af94914b56452243b07572a 2023-04-22 19:56:52,090 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/rep_barrier/8f3d96068af94914b56452243b07572a, entries=6, sequenceid=93, filesize=5.5 K 2023-04-22 19:56:52,091 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/.tmp/table/7fd4551af47e413499f7c6db870ad10d as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/table/7fd4551af47e413499f7c6db870ad10d 2023-04-22 19:56:52,093 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7fd4551af47e413499f7c6db870ad10d 2023-04-22 19:56:52,093 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/table/7fd4551af47e413499f7c6db870ad10d, entries=9, sequenceid=93, filesize=5.5 K 2023-04-22 19:56:52,094 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1244ms, sequenceid=93, compaction requested=false 2023-04-22 19:56:52,099 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-22 19:56:52,099 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-22 19:56:52,101 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-22 19:56:52,101 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-22 19:56:52,101 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-22 19:56:52,254 INFO [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44791,1682193215766; all regions closed. 2023-04-22 19:56:52,262 DEBUG [RS:0;jenkins-hbase4:44791] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/oldWALs 2023-04-22 19:56:52,262 INFO [RS:0;jenkins-hbase4:44791] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44791%2C1682193215766.meta:.meta(num 1682193218607) 2023-04-22 19:56:52,267 DEBUG [RS:0;jenkins-hbase4:44791] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/oldWALs 2023-04-22 19:56:52,267 INFO [RS:0;jenkins-hbase4:44791] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44791%2C1682193215766:(num 1682193217660) 2023-04-22 19:56:52,267 DEBUG [RS:0;jenkins-hbase4:44791] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:52,267 INFO [RS:0;jenkins-hbase4:44791] regionserver.LeaseManager(133): Closed leases 2023-04-22 19:56:52,267 INFO [RS:0;jenkins-hbase4:44791] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-22 19:56:52,268 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-22 19:56:52,269 INFO [RS:0;jenkins-hbase4:44791] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44791 2023-04-22 19:56:52,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 19:56:52,275 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44791,1682193215766 2023-04-22 19:56:52,275 ERROR [Listener at localhost/46665-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2ace51a5 rejected from java.util.concurrent.ThreadPoolExecutor@2ef1c024[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2296] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 19:56:52,276 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 19:56:52,276 ERROR [Listener at localhost/46665-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2c91ebf rejected from java.util.concurrent.ThreadPoolExecutor@2ef1c024[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2296] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 19:56:52,277 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,44791,1682193215766] 2023-04-22 19:56:52,278 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,44791,1682193215766; numProcessing=1 2023-04-22 19:56:52,279 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,44791,1682193215766 already deleted, retry=false 2023-04-22 19:56:52,279 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,44791,1682193215766 expired; onlineServers=0 2023-04-22 19:56:52,279 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34929,1682193214523' ***** 2023-04-22 19:56:52,279 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-22 19:56:52,279 DEBUG [M:0;jenkins-hbase4:34929] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13ccb7d1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-22 19:56:52,279 INFO [M:0;jenkins-hbase4:34929] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34929,1682193214523 2023-04-22 19:56:52,279 INFO [M:0;jenkins-hbase4:34929] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34929,1682193214523; all regions closed. 2023-04-22 19:56:52,279 DEBUG [M:0;jenkins-hbase4:34929] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 19:56:52,279 DEBUG [M:0;jenkins-hbase4:34929] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-22 19:56:52,279 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-22 19:56:52,279 DEBUG [M:0;jenkins-hbase4:34929] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-22 19:56:52,279 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682193217321] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682193217321,5,FailOnTimeoutGroup] 2023-04-22 19:56:52,279 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682193217325] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682193217325,5,FailOnTimeoutGroup] 2023-04-22 19:56:52,279 INFO [M:0;jenkins-hbase4:34929] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-22 19:56:52,280 INFO [M:0;jenkins-hbase4:34929] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-22 19:56:52,280 INFO [M:0;jenkins-hbase4:34929] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-22 19:56:52,280 DEBUG [M:0;jenkins-hbase4:34929] master.HMaster(1512): Stopping service threads 2023-04-22 19:56:52,280 INFO [M:0;jenkins-hbase4:34929] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-22 19:56:52,280 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-22 19:56:52,281 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 19:56:52,281 ERROR [M:0;jenkins-hbase4:34929] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (249600520) connection to localhost/127.0.0.1:32897 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-22 19:56:52,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-22 19:56:52,281 INFO [M:0;jenkins-hbase4:34929] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 19:56:52,281 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-22 19:56:52,282 DEBUG [M:0;jenkins-hbase4:34929] zookeeper.ZKUtil(398): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-22 19:56:52,282 WARN [M:0;jenkins-hbase4:34929] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-22 19:56:52,282 INFO [M:0;jenkins-hbase4:34929] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 19:56:52,282 INFO [M:0;jenkins-hbase4:34929] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-22 19:56:52,282 DEBUG [M:0;jenkins-hbase4:34929] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-22 19:56:52,283 INFO [M:0;jenkins-hbase4:34929] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 19:56:52,283 DEBUG [M:0;jenkins-hbase4:34929] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 19:56:52,283 DEBUG [M:0;jenkins-hbase4:34929] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-22 19:56:52,283 DEBUG [M:0;jenkins-hbase4:34929] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 19:56:52,283 INFO [M:0;jenkins-hbase4:34929] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.04 KB heapSize=393.72 KB 2023-04-22 19:56:52,292 INFO [M:0;jenkins-hbase4:34929] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.04 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/8a746a8340a247d4bf105083c7155c5b 2023-04-22 19:56:52,295 INFO [M:0;jenkins-hbase4:34929] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8a746a8340a247d4bf105083c7155c5b 2023-04-22 19:56:52,296 DEBUG [M:0;jenkins-hbase4:34929] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/8a746a8340a247d4bf105083c7155c5b as hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/8a746a8340a247d4bf105083c7155c5b 2023-04-22 19:56:52,299 INFO [M:0;jenkins-hbase4:34929] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8a746a8340a247d4bf105083c7155c5b 2023-04-22 19:56:52,299 INFO [M:0;jenkins-hbase4:34929] regionserver.HStore(1080): Added hdfs://localhost:32897/user/jenkins/test-data/715c54d7-a2c3-35f0-9dfb-4024baae9d04/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/8a746a8340a247d4bf105083c7155c5b, entries=65, sequenceid=578, filesize=24.7 K 2023-04-22 19:56:52,300 INFO [M:0;jenkins-hbase4:34929] regionserver.HRegion(2948): Finished flush of dataSize ~345.04 KB/353318, heapSize ~393.70 KB/403152, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 17ms, sequenceid=578, compaction requested=false 2023-04-22 19:56:52,301 INFO [M:0;jenkins-hbase4:34929] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 19:56:52,301 DEBUG [M:0;jenkins-hbase4:34929] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-22 19:56:52,304 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-22 19:56:52,304 INFO [M:0;jenkins-hbase4:34929] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-22 19:56:52,305 INFO [M:0;jenkins-hbase4:34929] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34929 2023-04-22 19:56:52,313 DEBUG [M:0;jenkins-hbase4:34929] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,34929,1682193214523 already deleted, retry=false 2023-04-22 19:56:52,378 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 19:56:52,378 INFO [RS:0;jenkins-hbase4:44791] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44791,1682193215766; zookeeper connection closed. 2023-04-22 19:56:52,378 ERROR [Listener at localhost/46665-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7a822fb4 rejected from java.util.concurrent.ThreadPoolExecutor@2ef1c024[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2296] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 19:56:52,379 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): regionserver:44791-0x101ad6f52b90001, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 19:56:52,379 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5c75eeaa] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5c75eeaa 2023-04-22 19:56:52,379 ERROR [Listener at localhost/46665-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@412a7ff3 rejected from java.util.concurrent.ThreadPoolExecutor@2ef1c024[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2296] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 19:56:52,380 INFO [Listener at localhost/46665] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-22 19:56:52,478 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 19:56:52,478 INFO [M:0;jenkins-hbase4:34929] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34929,1682193214523; zookeeper connection closed. 2023-04-22 19:56:52,478 ERROR [Listener at localhost/46665-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c2c60eb rejected from java.util.concurrent.ThreadPoolExecutor@3b17c56c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3837] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 19:56:52,479 DEBUG [Listener at localhost/46665-EventThread] zookeeper.ZKWatcher(600): master:34929-0x101ad6f52b90000, quorum=127.0.0.1:53381, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 19:56:52,479 ERROR [Listener at localhost/46665-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5afeb8e9 rejected from java.util.concurrent.ThreadPoolExecutor@3b17c56c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3837] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 19:56:52,480 WARN [Listener at localhost/46665] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-22 19:56:52,483 INFO [Listener at localhost/46665] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-22 19:56:52,591 WARN [BP-1192009695-172.31.14.131-1682193211166 heartbeating to localhost/127.0.0.1:32897] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-22 19:56:52,591 WARN [BP-1192009695-172.31.14.131-1682193211166 heartbeating to localhost/127.0.0.1:32897] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1192009695-172.31.14.131-1682193211166 (Datanode Uuid 2596931a-765d-4c8f-8b11-1321bc6bd088) service to localhost/127.0.0.1:32897 2023-04-22 19:56:52,595 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/cluster_cd12d633-6d44-ab55-c2cd-001d95f1c41f/dfs/data/data1/current/BP-1192009695-172.31.14.131-1682193211166] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-22 19:56:52,596 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f314b4e7-e69a-c315-f536-9d7f788f245c/cluster_cd12d633-6d44-ab55-c2cd-001d95f1c41f/dfs/data/data2/current/BP-1192009695-172.31.14.131-1682193211166] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-22 19:56:52,631 INFO [Listener at localhost/46665] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-22 19:56:52,794 INFO [Listener at localhost/46665] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-22 19:56:52,950 INFO [Listener at localhost/46665] hbase.HBaseTestingUtility(1293): Minicluster is down